Oct 30 12:48:57.590979 ip-10-0-130-200 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 12:48:57.590988 ip-10-0-130-200 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 12:48:57.590995 ip-10-0-130-200 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 12:48:57.591258 ip-10-0-130-200 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 12:49:07.840598 ip-10-0-130-200 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 12:49:07.840614 ip-10-0-130-200 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 53a4ba9e38784593b9d822433458753e -- Oct 30 12:51:05.995254 ip-10-0-130-200 systemd[1]: Starting Kubernetes Kubelet... Oct 30 12:51:06.358999 ip-10-0-130-200 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:51:06.358999 ip-10-0-130-200 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 12:51:06.358999 ip-10-0-130-200 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:51:06.358999 ip-10-0-130-200 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:51:06.363697 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363615 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:06.363697 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363697 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:06.363697 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363702 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363705 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363708 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363712 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363715 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363718 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363720 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363723 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363725 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363728 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363731 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363734 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363736 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363739 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363741 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363744 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363747 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363749 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363757 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363760 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:06.363798 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363763 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:06.364277 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363766 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:06.364277 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363768 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:06.364277 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363771 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:06.364277 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363774 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:06.364277 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363776 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:06.364277 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363779 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:06.364277 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363781 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:06.364277 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363784 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:06.364277 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363786 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:06.364277 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363789 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:06.364277 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363792 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:06.364277 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363795 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:06.364277 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363797 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:06.364277 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363801 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:06.364277 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363804 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:06.364277 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363808 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:06.364277 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363811 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:06.364277 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363814 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:06.364277 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363816 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363819 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363821 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363824 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363827 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363829 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363838 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363841 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363850 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363852 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363855 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363858 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363860 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363863 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363865 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363868 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363871 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363873 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363876 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363878 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:06.364730 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363881 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363884 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363886 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363889 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363891 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363894 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363896 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363900 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363902 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363905 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363908 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363910 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363913 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363915 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363918 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363920 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363923 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363927 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363930 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363956 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:06.365227 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363959 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:06.365692 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363962 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:06.365692 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363967 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:06.365692 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363971 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:06.365692 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363975 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:06.365692 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363978 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:06.365692 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363982 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:06.365692 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363984 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:06.365692 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363987 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:06.365692 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363990 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:06.365692 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363993 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:06.365692 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.363997 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:06.365692 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364000 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:06.365692 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364003 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:06.365692 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364006 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:06.365692 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364008 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:06.365692 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364010 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:06.365692 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364013 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:06.365692 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364018 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:06.365692 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364021 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364024 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364027 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364029 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364032 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364034 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364959 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364968 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364972 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364975 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364978 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364981 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364984 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364993 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364996 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.364999 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365002 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365004 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365006 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365021 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:06.366193 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365025 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365028 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365031 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365034 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365037 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365040 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365042 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365045 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365048 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365050 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365053 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365055 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365058 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365061 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365064 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365066 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365069 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365072 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365075 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365077 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:06.366706 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365080 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:06.367211 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365083 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:06.367211 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365085 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:06.367211 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365088 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:06.367211 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365091 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:06.367211 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365093 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:06.367211 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365097 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:06.367211 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365099 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:06.367211 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365102 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:06.367211 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365105 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:06.367211 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365107 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:06.367211 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365110 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:06.367211 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365112 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:06.367211 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365115 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:06.367211 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365118 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:06.367211 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365121 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:06.367211 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365124 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:06.367211 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365126 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:06.367211 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365129 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:06.367211 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365132 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365134 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365137 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365140 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365142 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365146 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365148 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365151 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365153 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365156 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365158 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365161 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365164 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365167 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365169 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365172 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365174 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365177 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365179 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365182 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:06.367659 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365185 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:06.368132 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365188 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:06.368132 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365190 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:06.368132 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365193 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:06.368132 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365195 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:06.368132 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365198 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:06.368132 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365203 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:06.368132 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365205 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:06.368132 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365208 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:06.368132 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365211 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:06.368132 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365213 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:06.368132 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365216 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:06.368132 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365218 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:06.368132 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365220 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:06.368132 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365224 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:06.368132 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365228 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:06.368132 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365230 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:06.368132 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365234 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:06.368132 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365237 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:06.368132 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365240 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:06.368578 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365243 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:06.368578 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365246 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:06.368578 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365248 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:06.368578 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365251 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:06.368578 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365254 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:06.368578 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365256 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:06.368578 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365261 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:06.368578 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365265 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:06.368578 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365268 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:06.368578 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365271 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:06.368578 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365274 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:06.368578 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365277 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:06.368578 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365279 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:06.368578 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365860 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:06.368578 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365865 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:06.368578 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365868 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:06.368578 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365871 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:06.368578 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365875 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:06.368578 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365880 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:06.369057 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365883 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:06.369057 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365886 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:06.369057 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365890 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:06.369057 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365893 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:06.369057 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365896 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:06.369057 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365899 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:06.369057 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365901 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:06.369057 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365904 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:06.369057 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365906 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:06.369057 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365909 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:06.369057 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365912 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:06.369057 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365915 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:06.369057 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365918 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:06.369057 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365920 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:06.369057 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365923 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:06.369057 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365925 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:06.369057 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365928 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:06.369057 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365930 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:06.369057 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365933 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365950 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365953 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365956 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365958 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365961 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365964 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365967 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365970 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365981 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365984 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365986 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365989 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365992 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365994 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.365997 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366000 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366003 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366005 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366008 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:06.369515 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366011 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:06.370009 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366014 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:06.370009 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366016 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:06.370009 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366019 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:06.370009 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366021 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:06.370009 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366024 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:06.370009 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366027 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:06.370009 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366029 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:06.370009 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366032 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:06.370009 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366035 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:06.370009 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366037 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:06.370009 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366040 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:06.370009 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366042 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:06.370009 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366045 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:06.370009 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366047 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:06.370009 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366050 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:06.370009 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366053 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:06.370009 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366056 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:06.370009 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366060 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:06.370009 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366063 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:06.370456 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366065 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:06.370456 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366069 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:06.370456 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366071 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:06.370456 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366074 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:06.370456 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366076 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:06.370456 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366079 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:06.370456 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366082 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:06.370456 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366086 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:06.370456 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366088 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:06.370456 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366091 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:06.370456 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366094 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:06.370456 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366096 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:06.370456 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366098 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:06.370456 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366101 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:06.370456 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366104 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:06.370456 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366106 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:06.370456 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366109 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:06.370456 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366111 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:06.370456 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366114 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366116 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366119 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366121 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366124 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366126 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366129 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366131 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366134 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366137 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366139 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366142 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366146 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366148 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366151 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366153 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366157 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366160 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366162 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366165 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:06.371105 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366167 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366170 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366173 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.366176 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367045 2581 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367061 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367070 2581 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367074 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367079 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367082 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367086 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367094 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367098 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367101 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367104 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367108 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367111 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367114 2581 flags.go:64] FLAG: --cgroup-root="" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367117 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367120 2581 flags.go:64] FLAG: --client-ca-file="" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367123 2581 flags.go:64] FLAG: --cloud-provider="external" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367126 2581 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367130 2581 flags.go:64] FLAG: --cluster-domain="" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367133 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 12:51:06.371580 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367138 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367141 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367145 2581 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367149 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367152 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367156 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367159 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367164 2581 flags.go:64] FLAG: --contention-profiling="false" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367167 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367170 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367174 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367177 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367181 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367185 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367188 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367191 2581 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367194 2581 flags.go:64] FLAG: --enable-server="true" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367197 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367202 2581 flags.go:64] FLAG: --event-burst="100" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367205 2581 flags.go:64] FLAG: --event-qps="50" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367208 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367211 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367214 2581 flags.go:64] FLAG: --eviction-hard="" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367219 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 12:51:06.372155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367222 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367225 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367228 2581 flags.go:64] FLAG: --eviction-soft="" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367232 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367235 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367238 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367241 2581 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367243 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367248 2581 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367251 2581 flags.go:64] FLAG: --feature-gates="" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367256 2581 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367259 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367262 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367266 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367269 2581 flags.go:64] FLAG: --healthz-port="10248" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367274 2581 flags.go:64] FLAG: --help="false" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367277 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-130-200.ec2.internal" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367280 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367284 2581 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367286 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367290 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367294 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367297 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 12:51:06.372743 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367300 2581 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367303 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367305 2581 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367309 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367312 2581 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367315 2581 flags.go:64] FLAG: --kube-reserved="" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367318 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367321 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367324 2581 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367327 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367329 2581 flags.go:64] FLAG: --lock-file="" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367332 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367335 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367338 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367345 2581 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367347 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367350 2581 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367355 2581 flags.go:64] FLAG: --logging-format="text" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367358 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367361 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367364 2581 flags.go:64] FLAG: --manifest-url="" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367367 2581 flags.go:64] FLAG: --manifest-url-header="" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367372 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367375 2581 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367381 2581 flags.go:64] FLAG: --max-pods="110" Oct 30 12:51:06.373306 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367385 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367389 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367392 2581 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367395 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367398 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367401 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367404 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367411 2581 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367415 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367418 2581 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367421 2581 flags.go:64] FLAG: --pod-cidr="" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367424 2581 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367427 2581 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367430 2581 flags.go:64] FLAG: --pods-per-core="0" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367433 2581 flags.go:64] FLAG: --port="10250" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367436 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367439 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cdf3b5499be48d54" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367442 2581 flags.go:64] FLAG: --qos-reserved="" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367445 2581 flags.go:64] FLAG: --read-only-port="10255" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367448 2581 flags.go:64] FLAG: --register-node="true" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367451 2581 flags.go:64] FLAG: --register-with-taints="" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367455 2581 flags.go:64] FLAG: --registry-burst="10" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367458 2581 flags.go:64] FLAG: --registry-qps="5" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367460 2581 flags.go:64] FLAG: --reserved-cpus="" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367464 2581 flags.go:64] FLAG: --reserved-memory="" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367468 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 12:51:06.373891 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367471 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367474 2581 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367477 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367480 2581 flags.go:64] FLAG: --runonce="false" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367483 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367488 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367491 2581 flags.go:64] FLAG: --seccomp-default="false" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367495 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367497 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367501 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367504 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367507 2581 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367510 2581 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367513 2581 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367516 2581 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367519 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367523 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367526 2581 flags.go:64] FLAG: --system-cgroups="" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367528 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367534 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367537 2581 flags.go:64] FLAG: --tls-cert-file="" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367540 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367544 2581 flags.go:64] FLAG: --tls-min-version="" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367547 2581 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367550 2581 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 12:51:06.374610 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367553 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367556 2581 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367559 2581 flags.go:64] FLAG: --v="2" Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367563 2581 flags.go:64] FLAG: --version="false" Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367567 2581 flags.go:64] FLAG: --vmodule="" Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367572 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.367576 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367721 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367725 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367728 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367731 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367734 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367738 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367741 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367744 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367747 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367749 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367752 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367755 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367758 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367760 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:06.375214 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367763 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367766 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367769 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367771 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367774 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367776 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367779 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367781 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367784 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367786 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367789 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367792 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367795 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367798 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367800 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367803 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367806 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367809 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367811 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367814 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:06.375713 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367817 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367819 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367822 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367826 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367828 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367831 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367834 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367836 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367839 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367842 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367844 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367847 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367849 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367852 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367856 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367860 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367863 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367866 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367868 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367871 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:06.376185 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367873 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:06.376661 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367876 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:06.376661 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367879 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:06.376661 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367881 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:06.376661 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367884 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:06.376661 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367886 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:06.376661 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367889 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:06.376661 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367891 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:06.376661 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367894 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:06.376661 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367897 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:06.376661 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367900 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:06.376661 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367902 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:06.376661 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367905 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:06.376661 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367908 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:06.376661 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367910 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:06.376661 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367914 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:06.376661 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367916 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:06.376661 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367919 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:06.376661 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367922 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:06.376661 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367925 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:06.377182 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367927 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:06.377182 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367930 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:06.377182 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367933 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:06.377182 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367947 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:06.377182 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367950 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:06.377182 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367953 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:06.377182 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367955 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:06.377182 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367958 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:06.377182 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367960 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:06.377182 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367963 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:06.377182 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367965 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:06.377182 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367968 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:06.377182 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367970 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:06.377182 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367973 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:06.377182 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367977 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:06.377182 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367981 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:06.377182 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367984 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:06.377182 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367987 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:06.377182 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367990 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:06.377618 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367993 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:06.377618 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367996 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:06.377618 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.367999 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:06.377618 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368001 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:06.377618 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368004 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:06.377618 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368007 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:06.377618 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368009 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:06.377618 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368012 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:06.377618 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368015 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:06.377618 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368018 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:06.377618 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368021 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:06.377618 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368023 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:06.377618 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368026 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:06.377618 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.368053 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:51:06.377618 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368095 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:06.377618 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368099 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:06.378014 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368102 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:06.378014 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368105 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:06.378014 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368107 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:06.378014 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368110 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:06.378014 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368113 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:06.378014 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368116 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:06.378014 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368118 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:06.378014 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368120 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:06.378014 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368123 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:06.378014 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368126 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:06.378014 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368128 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:06.378014 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368131 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:06.378014 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368133 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:06.378014 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368136 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:06.378014 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368138 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:06.378014 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368141 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:06.378014 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368144 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:06.378014 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368146 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:06.378014 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368149 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368152 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368154 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368157 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368160 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368162 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368167 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368169 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368172 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368174 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368177 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368179 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368182 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368185 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368188 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368190 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368193 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368195 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368197 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368200 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:06.378460 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368203 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368205 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368208 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368210 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368213 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368215 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368218 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368220 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368224 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368228 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368231 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368233 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368238 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368241 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368243 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368246 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368249 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368251 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368255 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368258 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:06.378925 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368260 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:06.379413 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368263 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:06.379413 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368267 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:06.379413 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368270 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:06.379413 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368272 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:06.379413 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368275 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:06.379413 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368278 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:06.379413 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368281 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:06.379413 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368283 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:06.379413 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368286 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:06.379413 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368288 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:06.379413 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368290 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:06.379413 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368293 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:06.379413 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368296 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:06.379413 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368298 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:06.379413 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368301 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:06.379413 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368303 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:06.379413 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368306 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:06.379413 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368308 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:06.379413 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368311 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:06.379889 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368314 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:06.379889 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368316 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:06.379889 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368320 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:06.379889 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368322 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:06.379889 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368326 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:06.379889 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368329 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:06.379889 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368331 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:06.379889 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368334 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:06.379889 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368337 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:06.379889 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368341 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:06.379889 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368345 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:06.379889 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368348 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:06.379889 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368351 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:06.379889 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368353 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:06.379889 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368356 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:06.379889 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368359 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:06.379889 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368361 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:06.379889 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368365 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:06.379889 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368368 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368370 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368373 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368375 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368378 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368380 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368383 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.368385 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.374972 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.374987 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.375517 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.375523 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375592 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375598 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375602 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375605 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375608 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375611 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375614 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375617 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:06.380354 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375620 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375622 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375625 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375628 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375631 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375634 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375636 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375639 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375642 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375644 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375647 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375650 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375658 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375661 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375664 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375667 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375669 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375672 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375675 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375678 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:06.380830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375681 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:06.381383 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375684 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:06.381383 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375687 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:06.381383 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375689 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:06.381383 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375692 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:06.381383 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375695 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:06.381383 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375697 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:06.381383 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375700 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:06.381383 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375704 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:06.381383 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375708 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:06.381383 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375711 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:06.381383 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375714 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:06.381383 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375717 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:06.381383 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375720 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:06.381383 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375723 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:06.381383 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375732 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:06.381383 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375735 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:06.381383 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375738 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:06.381383 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375740 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:06.381383 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375743 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375746 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375749 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375752 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375754 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375757 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375765 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375768 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375771 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375773 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375776 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375779 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375782 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375785 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375787 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375790 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375793 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375795 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375798 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375800 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:06.381830 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375803 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:06.382331 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375806 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:06.382331 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375808 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:06.382331 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375811 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:06.382331 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375813 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:06.382331 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375816 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:06.382331 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375818 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:06.382331 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375821 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:06.382331 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375823 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:06.382331 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375826 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:06.382331 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375829 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:06.382331 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375832 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:06.382331 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375834 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:06.382331 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375837 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:06.382331 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375840 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:06.382331 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375842 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:06.382331 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375845 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:06.382331 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375848 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:06.382331 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375850 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:06.382331 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375854 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:06.382773 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375856 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:06.382773 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375859 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:06.382773 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375862 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:06.382773 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375864 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:06.382773 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375867 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:06.382773 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375870 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:06.382773 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375872 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:06.382773 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375875 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:06.382773 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375877 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:06.382773 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375880 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:06.382773 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375882 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:06.382773 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375885 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:06.382773 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375888 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:06.382773 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375890 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:06.382773 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375894 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:06.382773 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375897 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:06.382773 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375900 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:06.382773 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375903 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:06.382773 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.375905 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:06.383292 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.375948 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:51:06.383292 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376121 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:06.383292 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376127 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:06.383292 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376131 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:06.383292 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376134 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:06.383292 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376137 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:06.383292 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376140 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:06.383292 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376143 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:06.383292 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376146 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:06.383292 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376148 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:06.383292 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376152 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:06.383292 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376155 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:06.383292 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376158 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:06.383292 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376162 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:06.383292 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376164 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:06.383292 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376167 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376170 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376172 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376175 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376177 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376180 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376182 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376185 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376187 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376190 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376193 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376196 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376198 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376201 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376203 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376206 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376209 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376212 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376214 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376217 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:06.383666 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376220 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:06.384161 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376223 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:06.384161 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376225 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:06.384161 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376228 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:06.384161 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376231 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:06.384161 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376233 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:06.384161 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376236 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:06.384161 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376238 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:06.384161 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376241 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:06.384161 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376243 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:06.384161 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376246 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:06.384161 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376249 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:06.384161 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376251 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:06.384161 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376254 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:06.384161 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376256 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:06.384161 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376259 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:06.384161 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376262 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:06.384161 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376264 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:06.384161 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376267 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:06.384161 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376269 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376271 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376274 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376277 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376279 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376282 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376284 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376287 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376289 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376292 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376294 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376296 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376299 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376301 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376304 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376307 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376309 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376312 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376315 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376317 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:06.384652 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376320 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:06.385926 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376322 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:06.385926 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376325 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:06.385926 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376327 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:06.385926 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376330 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:06.385926 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376332 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:06.385926 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376335 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:06.385926 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376337 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:06.385926 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376340 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:06.385926 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376342 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:06.385926 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376345 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:06.385926 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376347 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:06.385926 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376349 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:06.385926 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376352 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:06.385926 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376355 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:06.385926 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376357 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:06.385926 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376360 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:06.385926 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376362 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:06.385926 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376364 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:06.385926 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376367 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:06.386538 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376369 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:06.386538 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376372 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:06.386538 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376375 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:06.386538 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376377 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:06.386538 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376380 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:06.386538 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376384 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:06.386538 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376388 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:06.386538 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376391 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:06.386538 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376394 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:06.386538 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376397 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:06.386538 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376400 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:06.386538 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376403 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:06.386538 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376405 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:06.386538 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.376428 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:51:06.386538 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376461 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376464 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376467 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376470 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376474 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376476 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376479 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376481 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376484 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376487 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376490 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376493 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376495 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376498 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376500 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376503 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376505 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376508 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376511 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376513 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:06.386888 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376516 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376519 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376521 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376524 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376526 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376529 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376531 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376534 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376536 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376539 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376541 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376544 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376546 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376549 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376551 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376554 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376556 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376559 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376562 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376564 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:06.387468 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376567 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:06.388106 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376573 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:06.388106 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376577 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:06.388106 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376582 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:06.388106 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376585 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:06.388106 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376587 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:06.388106 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376590 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:06.388106 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376593 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:06.388106 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376595 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:06.388106 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376598 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:06.388106 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376600 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:06.388106 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376603 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:06.388106 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376606 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:06.388106 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376610 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:06.388106 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376612 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:06.388106 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376615 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:06.388106 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376618 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:06.388106 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376620 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:06.388106 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376624 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:06.388106 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376627 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376630 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376633 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376636 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376638 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376641 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376643 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376646 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376648 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376651 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376653 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376657 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376659 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376662 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376667 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376670 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376674 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376677 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376679 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376682 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:06.388641 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376685 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:06.389293 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376687 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:06.389293 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376690 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:06.389293 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376693 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:06.389293 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376696 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:06.389293 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376698 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:06.389293 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376701 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:06.389293 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376704 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:06.389293 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376706 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:06.389293 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376709 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:06.389293 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376711 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:06.389293 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376714 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:06.389293 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376716 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:06.389293 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376719 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:06.389293 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376721 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:06.389293 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376724 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:06.389293 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376726 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:06.389293 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376729 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:06.389293 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376731 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:06.389293 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376735 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:06.389751 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376738 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:06.389751 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376740 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:06.389751 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376743 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:06.389751 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376745 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:06.389751 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376752 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:06.389751 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376755 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:06.389751 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:06.376758 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:06.389751 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.376811 2581 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 12:51:06.389751 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.378687 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 12:51:06.389751 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.379420 2581 server.go:1014] "Starting client certificate rotation" Oct 30 12:51:06.389751 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.379516 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 12:51:06.389751 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.379553 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 12:51:06.389751 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.381263 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:51:06.389751 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.382915 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:51:06.396258 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.396241 2581 log.go:25] "Validated CRI v1 runtime API" Oct 30 12:51:06.399243 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.399227 2581 log.go:25] "Validated CRI v1 image API" Oct 30 12:51:06.400111 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.400087 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 12:51:06.405266 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.405241 2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c2c122e1-0639-4b96-aa24-52adb3b746a3:/dev/nvme0n1p4 d46bed61-ebbd-4ad9-97e3-86f7c526e1e8:/dev/nvme0n1p3] Oct 30 12:51:06.405319 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.405267 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 12:51:06.411081 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.410958 2581 manager.go:217] Machine: {Timestamp:2026-10-30 12:51:06.40962008 +0000 UTC m=+0.309649134 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3136542 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e7d27ba44568c9f4ff38fcaadefd8 SystemUUID:ec2e7d27-ba44-568c-9f4f-f38fcaadefd8 BootID:53a4ba9e-3878-4593-b9d8-22433458753e Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:97:5a:49 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:97:5a:49 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:e6:d0:f2:96:b6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 12:51:06.411081 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.411075 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 12:51:06.411219 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.411203 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 12:51:06.413355 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.413327 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 12:51:06.413503 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.413356 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-200.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 12:51:06.413549 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.413504 2581 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 12:51:06.413549 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.413512 2581 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 12:51:06.413598 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.413552 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:51:06.413598 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.413570 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:51:06.414697 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.414686 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 12:51:06.417185 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.417173 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 12:51:06.417305 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.417296 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 12:51:06.417883 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.417874 2581 kubelet.go:493] "Attempting to sync node with API server" Oct 30 12:51:06.417911 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.417888 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 12:51:06.417911 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.417899 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 12:51:06.417911 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.417908 2581 kubelet.go:405] "Adding apiserver pod source" Oct 30 12:51:06.418037 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.417916 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 12:51:06.418274 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.418260 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:51:06.420259 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.420244 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 12:51:06.423426 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.423413 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:51:06.424292 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.424279 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 12:51:06.424337 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.424310 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 12:51:06.424983 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.424972 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 12:51:06.425044 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.424990 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 12:51:06.425044 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.424999 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 12:51:06.425044 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.425007 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 12:51:06.425044 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.425015 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 12:51:06.425044 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.425020 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 12:51:06.425671 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.425661 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 12:51:06.425671 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.425671 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 12:51:06.425727 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.425678 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 12:51:06.425727 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.425684 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 12:51:06.425727 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.425692 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 12:51:06.425727 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.425702 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 12:51:06.426239 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.426229 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 12:51:06.426239 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.426239 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 12:51:06.429926 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.429913 2581 server.go:1263] "Started kubelet" Oct 30 12:51:06.430044 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.430010 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 12:51:06.430509 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.430240 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 12:51:06.430626 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.430611 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 12:51:06.430758 ip-10-0-130-200 systemd[1]: Started Kubernetes Kubelet. Oct 30 12:51:06.431789 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.431754 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 12:51:06.436084 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:06.436059 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 12:51:06.436781 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.436769 2581 server.go:317] "Adding debug handlers to kubelet server" Oct 30 12:51:06.437191 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.437167 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 12:51:06.437616 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.437597 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 12:51:06.437842 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:06.437817 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-200.ec2.internal\" not found" Oct 30 12:51:06.438235 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.438207 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 12:51:06.438235 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.438229 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 12:51:06.438374 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.438245 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 12:51:06.438434 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.438394 2581 reconstruct.go:97] "Volume reconstruction finished" Oct 30 12:51:06.438434 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.438404 2581 reconciler.go:29] "Reconciler: start to sync state" Oct 30 12:51:06.438749 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.438729 2581 factory.go:55] Registering systemd factory Oct 30 12:51:06.438822 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.438752 2581 factory.go:223] Registration of the systemd container factory successfully Oct 30 12:51:06.439010 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.438996 2581 factory.go:153] Registering CRI-O factory Oct 30 12:51:06.439059 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.439014 2581 factory.go:223] Registration of the crio container factory successfully Oct 30 12:51:06.439102 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.439071 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 12:51:06.439102 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.439090 2581 factory.go:103] Registering Raw factory Oct 30 12:51:06.439175 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.439104 2581 manager.go:1199] Started watching for new ooms in manager Oct 30 12:51:06.439970 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.439934 2581 manager.go:319] Starting recovery of all containers Oct 30 12:51:06.447775 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.447758 2581 manager.go:324] Recovery completed Oct 30 12:51:06.452301 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.452283 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:06.454269 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.454240 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:06.454344 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.454287 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:06.454344 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.454301 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:06.454860 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.454844 2581 cpu_manager.go:226] "Starting" policy="none" Oct 30 12:51:06.454860 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.454856 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:06.455004 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.454857 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 12:51:06.455004 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.454900 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 12:51:06.455496 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.455481 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:06.455543 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.455508 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:06.455543 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.455522 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:06.456332 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.456318 2581 policy_none.go:50] "Start" Oct 30 12:51:06.456400 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.456336 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 12:51:06.456400 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.456349 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 12:51:06.457354 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.457339 2581 policy_none.go:44] "Start" Oct 30 12:51:06.464106 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:06.464081 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-200.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 12:51:06.465866 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.465825 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-200.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 12:51:06.474143 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:06.473164 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-200.ec2.internal.18e34fa9974a4d8e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-200.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-200.ec2.internal,},FirstTimestamp:2026-10-30 12:51:06.429885838 +0000 UTC m=+0.329914892,LastTimestamp:2026-10-30 12:51:06.429885838 +0000 UTC m=+0.329914892,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-200.ec2.internal,}" Oct 30 12:51:06.478919 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:06.478892 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 12:51:06.479038 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:06.478999 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 12:51:06.483290 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.483260 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 12:51:06.485731 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.485718 2581 manager.go:341] "Starting Device Plugin manager" Oct 30 12:51:06.485815 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:06.485746 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 12:51:06.485815 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.485753 2581 server.go:87] "Starting device plugin registration server" Oct 30 12:51:06.486034 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.486023 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 12:51:06.486067 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.486035 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 12:51:06.486285 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.486147 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 12:51:06.486285 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.486230 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 12:51:06.486285 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.486238 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 12:51:06.486722 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:06.486693 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 12:51:06.486805 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:06.486747 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-200.ec2.internal\" not found" Oct 30 12:51:06.488961 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.488929 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 12:51:06.489043 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.489017 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6zhj8" Oct 30 12:51:06.494706 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.494689 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6zhj8" Oct 30 12:51:06.504875 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:06.504836 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-200.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 12:51:06.504998 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:06.504911 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-200.ec2.internal.18e34fa998be5b8d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-200.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-200.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-200.ec2.internal,},FirstTimestamp:2026-10-30 12:51:06.454268813 +0000 UTC m=+0.354297874,LastTimestamp:2026-10-30 12:51:06.454268813 +0000 UTC m=+0.354297874,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-200.ec2.internal,}" Oct 30 12:51:06.560600 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.560568 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 12:51:06.561869 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.561850 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 12:51:06.561979 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.561878 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 12:51:06.561979 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.561903 2581 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 12:51:06.561979 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:06.561959 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 12:51:06.579151 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.579124 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:06.586204 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.586183 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:06.587256 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.587234 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:06.587351 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.587274 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:06.587351 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.587291 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:06.587351 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.587322 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-200.ec2.internal" Oct 30 12:51:06.606548 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.606523 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-200.ec2.internal" Oct 30 12:51:06.606548 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:06.606552 2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-200.ec2.internal\": node \"ip-10-0-130-200.ec2.internal\" not found" Oct 30 12:51:06.647230 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:06.647142 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-200.ec2.internal\" not found" Oct 30 12:51:06.662870 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.662840 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal"] Oct 30 12:51:06.662965 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.662907 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:06.664044 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.664017 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:06.664044 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.664048 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:06.664210 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.664062 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:06.665478 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.665458 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:06.665630 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.665612 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" Oct 30 12:51:06.665692 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.665652 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:06.666300 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.666283 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:06.666402 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.666311 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:06.666402 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.666326 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:06.666402 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.666289 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:06.666402 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.666379 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:06.666402 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.666388 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:06.667658 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.667644 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal" Oct 30 12:51:06.667707 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.667668 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:06.668351 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.668335 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:06.668423 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.668366 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:06.668423 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.668378 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:06.692774 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:06.692749 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-200.ec2.internal\" not found" node="ip-10-0-130-200.ec2.internal" Oct 30 12:51:06.696073 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:06.696055 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-200.ec2.internal\" not found" node="ip-10-0-130-200.ec2.internal" Oct 30 12:51:06.739872 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.739846 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba36f1221f97bdfc5568490402fa51dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal\" (UID: \"ba36f1221f97bdfc5568490402fa51dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" Oct 30 12:51:06.739872 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.739875 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1dd3ede18217038ba719ab8e2dbe78f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-200.ec2.internal\" (UID: \"f1dd3ede18217038ba719ab8e2dbe78f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal" Oct 30 12:51:06.740081 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.739891 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba36f1221f97bdfc5568490402fa51dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal\" (UID: \"ba36f1221f97bdfc5568490402fa51dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" Oct 30 12:51:06.747976 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:06.747953 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-200.ec2.internal\" not found" Oct 30 12:51:06.840046 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.840020 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba36f1221f97bdfc5568490402fa51dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal\" (UID: \"ba36f1221f97bdfc5568490402fa51dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" Oct 30 12:51:06.840115 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.840053 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba36f1221f97bdfc5568490402fa51dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal\" (UID: \"ba36f1221f97bdfc5568490402fa51dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" Oct 30 12:51:06.840115 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.840070 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1dd3ede18217038ba719ab8e2dbe78f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-200.ec2.internal\" (UID: \"f1dd3ede18217038ba719ab8e2dbe78f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal" Oct 30 12:51:06.840186 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.840153 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba36f1221f97bdfc5568490402fa51dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal\" (UID: \"ba36f1221f97bdfc5568490402fa51dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" Oct 30 12:51:06.840186 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.840180 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1dd3ede18217038ba719ab8e2dbe78f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-200.ec2.internal\" (UID: \"f1dd3ede18217038ba719ab8e2dbe78f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal" Oct 30 12:51:06.840259 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.840208 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba36f1221f97bdfc5568490402fa51dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal\" (UID: \"ba36f1221f97bdfc5568490402fa51dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" Oct 30 12:51:06.848170 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:06.848146 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-200.ec2.internal\" not found" Oct 30 12:51:06.948883 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:06.948805 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-200.ec2.internal\" not found" Oct 30 12:51:06.995350 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.995306 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" Oct 30 12:51:06.998270 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:06.998251 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal" Oct 30 12:51:07.049583 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:07.049551 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-200.ec2.internal\" not found" Oct 30 12:51:07.150068 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:07.150035 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-200.ec2.internal\" not found" Oct 30 12:51:07.250643 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:07.250561 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-200.ec2.internal\" not found" Oct 30 12:51:07.336771 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:07.336730 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:07.351158 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:07.351132 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-200.ec2.internal\" not found" Oct 30 12:51:07.379427 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:07.379392 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 12:51:07.379867 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:07.379565 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 12:51:07.379867 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:07.379597 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 12:51:07.451999 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:07.451794 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-200.ec2.internal\" not found" Oct 30 12:51:07.498618 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:07.498553 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:46:06 +0000 UTC" deadline="2028-07-04 15:56:24.755219375 +0000 UTC" Oct 30 12:51:07.498618 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:07.498612 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14715h5m17.25661102s" Oct 30 12:51:07.530300 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:07.530258 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1dd3ede18217038ba719ab8e2dbe78f.slice/crio-f9217d2105f8c59ba473a15b0058a619e3a3de93310651204fc5c42c6dab6dc8 WatchSource:0}: Error finding container f9217d2105f8c59ba473a15b0058a619e3a3de93310651204fc5c42c6dab6dc8: Status 404 returned error can't find the container with id f9217d2105f8c59ba473a15b0058a619e3a3de93310651204fc5c42c6dab6dc8 Oct 30 12:51:07.530516 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:07.530493 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba36f1221f97bdfc5568490402fa51dc.slice/crio-52bf0edd4be66970dd5081a5a0f4864dec438a44a47223a8155d2a4541e8644f WatchSource:0}: Error finding container 52bf0edd4be66970dd5081a5a0f4864dec438a44a47223a8155d2a4541e8644f: Status 404 returned error can't find the container with id 52bf0edd4be66970dd5081a5a0f4864dec438a44a47223a8155d2a4541e8644f Oct 30 12:51:07.538647 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:07.538623 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:51:07.552333 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:07.552306 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-200.ec2.internal\" not found" Oct 30 12:51:07.566155 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:07.566097 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" event={"ID":"ba36f1221f97bdfc5568490402fa51dc","Type":"ContainerStarted","Data":"52bf0edd4be66970dd5081a5a0f4864dec438a44a47223a8155d2a4541e8644f"} Oct 30 12:51:07.567334 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:07.567302 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal" event={"ID":"f1dd3ede18217038ba719ab8e2dbe78f","Type":"ContainerStarted","Data":"f9217d2105f8c59ba473a15b0058a619e3a3de93310651204fc5c42c6dab6dc8"} Oct 30 12:51:07.639907 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:07.639874 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:07.738892 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:07.738854 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" Oct 30 12:51:07.745575 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:07.745549 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:07.765169 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:07.765145 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:51:07.766180 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:07.766136 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal" Oct 30 12:51:07.789736 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:07.789691 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:51:08.419391 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.419350 2581 apiserver.go:52] "Watching apiserver" Oct 30 12:51:08.439253 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.439207 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 12:51:08.466818 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.466783 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 12:51:08.467245 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.467211 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-z9f5d","openshift-network-diagnostics/network-check-target-xb4w4","openshift-dns/node-resolver-z6m2l","openshift-multus/multus-8l62c","openshift-multus/network-metrics-daemon-7l5g4","kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal","openshift-network-operator/iptables-alerter-4rj8r","openshift-ovn-kubernetes/ovnkube-node-k2v62","kube-system/global-pull-secret-syncer-r5lrn","kube-system/konnectivity-agent-pz66k","openshift-image-registry/node-ca-kvsqc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal","openshift-multus/multus-additional-cni-plugins-cl4bq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49"] Oct 30 12:51:08.468979 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.468802 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4rj8r" Oct 30 12:51:08.470547 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.470522 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.471647 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.471619 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z6m2l" Oct 30 12:51:08.472939 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.472918 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.474128 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.474064 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:08.474216 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:08.474137 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l5g4" podUID="6b553c14-1755-44ed-8dec-a2d1d8259c1d" Oct 30 12:51:08.476833 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.476314 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.477640 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.477618 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:08.477742 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:08.477683 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5lrn" podUID="87f27309-51ca-4116-b0e9-10cf9e144c36" Oct 30 12:51:08.478929 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.478908 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.480320 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.480297 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:08.480495 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:08.480372 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb4w4" podUID="30b22e49-8949-4dc1-9886-cc25b97f3436" Oct 30 12:51:08.481689 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.481668 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pz66k" Oct 30 12:51:08.483253 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.483232 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kvsqc" Oct 30 12:51:08.484850 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.484817 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.487802 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.487780 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 12:51:08.487909 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.487780 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:08.487909 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.487783 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 12:51:08.487909 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.487853 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:08.487909 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.487900 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 12:51:08.488140 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.487975 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 12:51:08.488140 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.488021 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-48745\"" Oct 30 12:51:08.488347 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.488235 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xj7mb\"" Oct 30 12:51:08.508020 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.507991 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 12:51:08.508811 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.508792 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 12:51:08.517368 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.517287 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d7fkx" Oct 30 12:51:08.533944 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.533920 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d7fkx" Oct 30 12:51:08.539662 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.539641 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 12:51:08.551132 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551103 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrvbt\" (UniqueName: \"kubernetes.io/projected/f35c9e60-a8e9-43b2-bea6-614f6555bc03-kube-api-access-vrvbt\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.551264 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551144 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-run-systemd\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.551264 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551175 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-run-openvswitch\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.551264 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551209 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-system-cni-dir\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.551264 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551236 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs\") pod \"network-metrics-daemon-7l5g4\" (UID: \"6b553c14-1755-44ed-8dec-a2d1d8259c1d\") " pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:08.551264 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551262 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-sysctl-d\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.551457 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551286 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-var-lib-kubelet\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.551457 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551310 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndhbb\" (UniqueName: \"kubernetes.io/projected/10b4d740-fb6d-486e-b6c1-5c6115a10496-kube-api-access-ndhbb\") pod \"node-ca-kvsqc\" (UID: \"10b4d740-fb6d-486e-b6c1-5c6115a10496\") " pod="openshift-image-registry/node-ca-kvsqc" Oct 30 12:51:08.551457 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551339 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c10fc6a1-af69-48c2-b3dc-131e48f36b22-hosts-file\") pod \"node-resolver-z6m2l\" (UID: \"c10fc6a1-af69-48c2-b3dc-131e48f36b22\") " pod="openshift-dns/node-resolver-z6m2l" Oct 30 12:51:08.551457 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551363 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-cni-binary-copy\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.551457 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551387 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-run-netns\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.551457 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551406 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-etc-kubernetes\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.551457 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551424 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-sys-fs\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.551457 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551438 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-run-netns\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.551457 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551453 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-node-log\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.551771 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551468 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-hostroot\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.551771 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551484 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-tuned\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.551771 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551500 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-kubelet\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.551771 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551516 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct5kf\" (UniqueName: \"kubernetes.io/projected/fc2355fd-e26c-4d96-b399-5528863ba211-kube-api-access-ct5kf\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.551771 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551532 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-host\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.551771 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551547 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:08.551771 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551563 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-cni-netd\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.551771 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551577 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-ovnkube-script-lib\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.551771 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551593 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-systemd\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.551771 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551607 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-sys\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.551771 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551630 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87f27309-51ca-4116-b0e9-10cf9e144c36-kubelet-config\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:08.551771 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551652 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lfnw\" (UniqueName: \"kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw\") pod \"network-check-target-xb4w4\" (UID: \"30b22e49-8949-4dc1-9886-cc25b97f3436\") " pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:08.551771 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551675 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc2355fd-e26c-4d96-b399-5528863ba211-cnibin\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.551771 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551702 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-var-lib-cni-bin\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.551771 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551727 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10b4d740-fb6d-486e-b6c1-5c6115a10496-serviceca\") pod \"node-ca-kvsqc\" (UID: \"10b4d740-fb6d-486e-b6c1-5c6115a10496\") " pod="openshift-image-registry/node-ca-kvsqc" Oct 30 12:51:08.551771 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551742 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51f1ecdc-2207-4ac7-9193-77f917cc0143-konnectivity-ca\") pod \"konnectivity-agent-pz66k\" (UID: \"51f1ecdc-2207-4ac7-9193-77f917cc0143\") " pod="kube-system/konnectivity-agent-pz66k" Oct 30 12:51:08.552405 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551771 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7gmw\" (UniqueName: \"kubernetes.io/projected/6b553c14-1755-44ed-8dec-a2d1d8259c1d-kube-api-access-f7gmw\") pod \"network-metrics-daemon-7l5g4\" (UID: \"6b553c14-1755-44ed-8dec-a2d1d8259c1d\") " pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:08.552405 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551796 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-sysctl-conf\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.552405 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551827 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c10fc6a1-af69-48c2-b3dc-131e48f36b22-tmp-dir\") pod \"node-resolver-z6m2l\" (UID: \"c10fc6a1-af69-48c2-b3dc-131e48f36b22\") " pod="openshift-dns/node-resolver-z6m2l" Oct 30 12:51:08.552405 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551855 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-device-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.552405 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551881 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpmwj\" (UniqueName: \"kubernetes.io/projected/061dda5b-09a3-493d-9c16-6d25055d5b82-kube-api-access-qpmwj\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.552405 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551898 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49rmt\" (UniqueName: \"kubernetes.io/projected/c10fc6a1-af69-48c2-b3dc-131e48f36b22-kube-api-access-49rmt\") pod \"node-resolver-z6m2l\" (UID: \"c10fc6a1-af69-48c2-b3dc-131e48f36b22\") " pod="openshift-dns/node-resolver-z6m2l" Oct 30 12:51:08.552405 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551915 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc2355fd-e26c-4d96-b399-5528863ba211-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.552405 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551935 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-modprobe-d\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.552405 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551958 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-run\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.552405 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.551988 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-registration-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.552405 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552010 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-var-lib-openvswitch\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.552405 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552048 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-ovnkube-config\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.552405 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552076 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-ovn-node-metrics-cert\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.552405 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552102 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tqpc\" (UniqueName: \"kubernetes.io/projected/e7af4112-7294-4d61-95bd-d05625f88bdc-kube-api-access-9tqpc\") pod \"iptables-alerter-4rj8r\" (UID: \"e7af4112-7294-4d61-95bd-d05625f88bdc\") " pod="openshift-network-operator/iptables-alerter-4rj8r" Oct 30 12:51:08.552405 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552128 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-var-lib-cni-multus\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.552405 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552153 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-var-lib-kubelet\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.552956 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552182 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-cni-bin\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.552956 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552211 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc2355fd-e26c-4d96-b399-5528863ba211-cni-binary-copy\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.552956 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552232 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-conf-dir\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.552956 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552258 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10b4d740-fb6d-486e-b6c1-5c6115a10496-host\") pod \"node-ca-kvsqc\" (UID: \"10b4d740-fb6d-486e-b6c1-5c6115a10496\") " pod="openshift-image-registry/node-ca-kvsqc" Oct 30 12:51:08.552956 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552283 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51f1ecdc-2207-4ac7-9193-77f917cc0143-agent-certs\") pod \"konnectivity-agent-pz66k\" (UID: \"51f1ecdc-2207-4ac7-9193-77f917cc0143\") " pod="kube-system/konnectivity-agent-pz66k" Oct 30 12:51:08.552956 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552306 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-slash\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.552956 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552321 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-cnibin\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.552956 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552343 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f35c9e60-a8e9-43b2-bea6-614f6555bc03-tmp\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.552956 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552360 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-etc-selinux\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.552956 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552376 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-systemd-units\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.552956 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552391 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-run-ovn\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.552956 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552408 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc2355fd-e26c-4d96-b399-5528863ba211-system-cni-dir\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.552956 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552434 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc2355fd-e26c-4d96-b399-5528863ba211-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.552956 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552459 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-run-multus-certs\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.552956 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552477 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-kubernetes\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.552956 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552503 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-lib-modules\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.552956 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552546 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-etc-openvswitch\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.553649 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552567 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e7af4112-7294-4d61-95bd-d05625f88bdc-iptables-alerter-script\") pod \"iptables-alerter-4rj8r\" (UID: \"e7af4112-7294-4d61-95bd-d05625f88bdc\") " pod="openshift-network-operator/iptables-alerter-4rj8r" Oct 30 12:51:08.553649 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552585 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7af4112-7294-4d61-95bd-d05625f88bdc-host-slash\") pod \"iptables-alerter-4rj8r\" (UID: \"e7af4112-7294-4d61-95bd-d05625f88bdc\") " pod="openshift-network-operator/iptables-alerter-4rj8r" Oct 30 12:51:08.553649 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552601 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-cni-dir\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.553649 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552616 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-socket-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.553649 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552635 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87f27309-51ca-4116-b0e9-10cf9e144c36-dbus\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:08.553649 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552654 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-log-socket\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.553649 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552692 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.553649 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552722 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.553649 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552739 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-env-overrides\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.553649 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552755 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6vzs\" (UniqueName: \"kubernetes.io/projected/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-kube-api-access-v6vzs\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.553649 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552772 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc2355fd-e26c-4d96-b399-5528863ba211-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.553649 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552788 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.553649 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552806 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-socket-dir-parent\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.553649 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552825 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-run-k8s-cni-cncf-io\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.553649 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552841 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-daemon-config\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.553649 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552859 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j76cj\" (UniqueName: \"kubernetes.io/projected/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-kube-api-access-j76cj\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.554317 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.552889 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-sysconfig\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.596699 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.596665 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 12:51:08.596896 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.596820 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 12:51:08.596896 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.596848 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 12:51:08.597052 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.596934 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 12:51:08.597052 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.596960 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 12:51:08.597052 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.597001 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 12:51:08.597052 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.597009 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 12:51:08.597052 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.597023 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:08.597052 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.597023 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 12:51:08.597350 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.597070 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 12:51:08.597350 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.597018 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:08.597350 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.597099 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 12:51:08.597350 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.597103 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 12:51:08.597350 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.597115 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 12:51:08.597350 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.597124 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 12:51:08.597350 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.597159 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 12:51:08.597350 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.597162 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qjjfj\"" Oct 30 12:51:08.597350 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.597194 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-87w8v\"" Oct 30 12:51:08.597350 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.597261 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 12:51:08.597350 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.597303 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 12:51:08.597350 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.597338 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-sfhnn\"" Oct 30 12:51:08.597801 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.597444 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m2lrf\"" Oct 30 12:51:08.610971 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.610934 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9ppdm\"" Oct 30 12:51:08.611150 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.611029 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 12:51:08.611150 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.611100 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7j54b\"" Oct 30 12:51:08.611150 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.611031 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-v4xk8\"" Oct 30 12:51:08.611150 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.611118 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 12:51:08.653619 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.653564 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:08.653619 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.653625 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-cni-netd\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.653866 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.653649 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-ovnkube-script-lib\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.653866 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.653675 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-systemd\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.653866 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.653699 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-sys\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.653866 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.653725 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87f27309-51ca-4116-b0e9-10cf9e144c36-kubelet-config\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:08.653866 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.653753 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfnw\" (UniqueName: \"kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw\") pod \"network-check-target-xb4w4\" (UID: \"30b22e49-8949-4dc1-9886-cc25b97f3436\") " pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:08.653866 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:08.653776 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:08.653866 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.653791 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-systemd\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.653866 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.653818 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc2355fd-e26c-4d96-b399-5528863ba211-cnibin\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.653866 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.653858 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-cni-netd\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.653866 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:08.653869 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret podName:87f27309-51ca-4116-b0e9-10cf9e144c36 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:09.153842649 +0000 UTC m=+3.053871719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret") pod "global-pull-secret-syncer-r5lrn" (UID: "87f27309-51ca-4116-b0e9-10cf9e144c36") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:08.654236 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.653870 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-sys\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.654236 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.653780 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc2355fd-e26c-4d96-b399-5528863ba211-cnibin\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.654236 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.653907 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87f27309-51ca-4116-b0e9-10cf9e144c36-kubelet-config\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:08.654236 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.653945 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-var-lib-cni-bin\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.654236 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.653983 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10b4d740-fb6d-486e-b6c1-5c6115a10496-serviceca\") pod \"node-ca-kvsqc\" (UID: \"10b4d740-fb6d-486e-b6c1-5c6115a10496\") " pod="openshift-image-registry/node-ca-kvsqc" Oct 30 12:51:08.654236 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.654025 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51f1ecdc-2207-4ac7-9193-77f917cc0143-konnectivity-ca\") pod \"konnectivity-agent-pz66k\" (UID: \"51f1ecdc-2207-4ac7-9193-77f917cc0143\") " pod="kube-system/konnectivity-agent-pz66k" Oct 30 12:51:08.654236 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.654059 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7gmw\" (UniqueName: \"kubernetes.io/projected/6b553c14-1755-44ed-8dec-a2d1d8259c1d-kube-api-access-f7gmw\") pod \"network-metrics-daemon-7l5g4\" (UID: \"6b553c14-1755-44ed-8dec-a2d1d8259c1d\") " pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:08.654236 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.654090 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-sysctl-conf\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.654468 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.654242 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-sysctl-conf\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.654468 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.654280 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c10fc6a1-af69-48c2-b3dc-131e48f36b22-tmp-dir\") pod \"node-resolver-z6m2l\" (UID: \"c10fc6a1-af69-48c2-b3dc-131e48f36b22\") " pod="openshift-dns/node-resolver-z6m2l" Oct 30 12:51:08.654468 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.654307 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-device-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.654468 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.654333 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpmwj\" (UniqueName: \"kubernetes.io/projected/061dda5b-09a3-493d-9c16-6d25055d5b82-kube-api-access-qpmwj\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.654468 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.654418 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-ovnkube-script-lib\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.654468 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.654448 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10b4d740-fb6d-486e-b6c1-5c6115a10496-serviceca\") pod \"node-ca-kvsqc\" (UID: \"10b4d740-fb6d-486e-b6c1-5c6115a10496\") " pod="openshift-image-registry/node-ca-kvsqc" Oct 30 12:51:08.654647 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.654500 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-var-lib-cni-bin\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.654647 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.654603 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-device-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.654706 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.654646 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49rmt\" (UniqueName: \"kubernetes.io/projected/c10fc6a1-af69-48c2-b3dc-131e48f36b22-kube-api-access-49rmt\") pod \"node-resolver-z6m2l\" (UID: \"c10fc6a1-af69-48c2-b3dc-131e48f36b22\") " pod="openshift-dns/node-resolver-z6m2l" Oct 30 12:51:08.654706 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.654646 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c10fc6a1-af69-48c2-b3dc-131e48f36b22-tmp-dir\") pod \"node-resolver-z6m2l\" (UID: \"c10fc6a1-af69-48c2-b3dc-131e48f36b22\") " pod="openshift-dns/node-resolver-z6m2l" Oct 30 12:51:08.654782 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.654707 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc2355fd-e26c-4d96-b399-5528863ba211-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.654834 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.654820 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-modprobe-d\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.654935 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.654912 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-run\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.655006 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.654968 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-registration-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.655006 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.654997 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-var-lib-openvswitch\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.655137 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655024 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-modprobe-d\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.655137 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655044 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51f1ecdc-2207-4ac7-9193-77f917cc0143-konnectivity-ca\") pod \"konnectivity-agent-pz66k\" (UID: \"51f1ecdc-2207-4ac7-9193-77f917cc0143\") " pod="kube-system/konnectivity-agent-pz66k" Oct 30 12:51:08.655137 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655062 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-var-lib-openvswitch\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.655137 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655086 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-ovnkube-config\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.655137 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655097 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-registration-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.655137 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655128 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-ovn-node-metrics-cert\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.655452 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655155 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-run\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.655452 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655160 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tqpc\" (UniqueName: \"kubernetes.io/projected/e7af4112-7294-4d61-95bd-d05625f88bdc-kube-api-access-9tqpc\") pod \"iptables-alerter-4rj8r\" (UID: \"e7af4112-7294-4d61-95bd-d05625f88bdc\") " pod="openshift-network-operator/iptables-alerter-4rj8r" Oct 30 12:51:08.655452 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655225 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-var-lib-cni-multus\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.655452 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655276 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-var-lib-kubelet\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.655452 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655306 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-cni-bin\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.655452 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655349 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc2355fd-e26c-4d96-b399-5528863ba211-cni-binary-copy\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.655452 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655400 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-cni-bin\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.655452 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655411 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc2355fd-e26c-4d96-b399-5528863ba211-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.655452 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655438 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-var-lib-cni-multus\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.655876 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655480 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-var-lib-kubelet\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.655876 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655518 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-conf-dir\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.655876 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655551 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10b4d740-fb6d-486e-b6c1-5c6115a10496-host\") pod \"node-ca-kvsqc\" (UID: \"10b4d740-fb6d-486e-b6c1-5c6115a10496\") " pod="openshift-image-registry/node-ca-kvsqc" Oct 30 12:51:08.655876 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655578 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51f1ecdc-2207-4ac7-9193-77f917cc0143-agent-certs\") pod \"konnectivity-agent-pz66k\" (UID: \"51f1ecdc-2207-4ac7-9193-77f917cc0143\") " pod="kube-system/konnectivity-agent-pz66k" Oct 30 12:51:08.655876 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655603 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-slash\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.655876 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655629 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-cnibin\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.655876 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655640 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-ovnkube-config\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.655876 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655652 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f35c9e60-a8e9-43b2-bea6-614f6555bc03-tmp\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.655876 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655725 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-etc-selinux\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.655876 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655749 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-systemd-units\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.655876 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655770 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-run-ovn\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.655876 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655792 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc2355fd-e26c-4d96-b399-5528863ba211-system-cni-dir\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.655876 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655817 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc2355fd-e26c-4d96-b399-5528863ba211-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.655876 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655841 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-conf-dir\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.655876 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655844 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-run-multus-certs\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.655876 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655882 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-run-multus-certs\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.656600 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655886 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-kubernetes\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.656600 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655911 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-slash\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.656600 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655913 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-lib-modules\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.656600 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655937 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-etc-openvswitch\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.656600 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655954 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e7af4112-7294-4d61-95bd-d05625f88bdc-iptables-alerter-script\") pod \"iptables-alerter-4rj8r\" (UID: \"e7af4112-7294-4d61-95bd-d05625f88bdc\") " pod="openshift-network-operator/iptables-alerter-4rj8r" Oct 30 12:51:08.656600 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655971 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7af4112-7294-4d61-95bd-d05625f88bdc-host-slash\") pod \"iptables-alerter-4rj8r\" (UID: \"e7af4112-7294-4d61-95bd-d05625f88bdc\") " pod="openshift-network-operator/iptables-alerter-4rj8r" Oct 30 12:51:08.656600 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655988 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-lib-modules\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.656600 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656004 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc2355fd-e26c-4d96-b399-5528863ba211-cni-binary-copy\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.656600 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656023 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10b4d740-fb6d-486e-b6c1-5c6115a10496-host\") pod \"node-ca-kvsqc\" (UID: \"10b4d740-fb6d-486e-b6c1-5c6115a10496\") " pod="openshift-image-registry/node-ca-kvsqc" Oct 30 12:51:08.656600 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656003 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 12:51:08.656600 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656081 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-kubernetes\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.656600 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656088 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-systemd-units\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.656600 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656118 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-cni-dir\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.656600 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656117 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-etc-openvswitch\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.656600 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656129 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-run-ovn\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.656600 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656144 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc2355fd-e26c-4d96-b399-5528863ba211-system-cni-dir\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.656600 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.655997 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-cni-dir\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.656600 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656183 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-socket-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.657495 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656189 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-cnibin\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.657495 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656205 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87f27309-51ca-4116-b0e9-10cf9e144c36-dbus\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:08.657495 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656265 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc2355fd-e26c-4d96-b399-5528863ba211-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.657495 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656267 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-etc-selinux\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.657495 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656269 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-log-socket\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.657495 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656292 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-log-socket\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.657495 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656304 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.657495 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656315 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87f27309-51ca-4116-b0e9-10cf9e144c36-dbus\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:08.657495 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656331 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.657495 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656350 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-env-overrides\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.657495 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656354 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.657495 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656366 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vzs\" (UniqueName: \"kubernetes.io/projected/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-kube-api-access-v6vzs\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.657495 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656373 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-socket-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.657495 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656399 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7af4112-7294-4d61-95bd-d05625f88bdc-host-slash\") pod \"iptables-alerter-4rj8r\" (UID: \"e7af4112-7294-4d61-95bd-d05625f88bdc\") " pod="openshift-network-operator/iptables-alerter-4rj8r" Oct 30 12:51:08.657495 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656406 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.657495 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656495 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc2355fd-e26c-4d96-b399-5528863ba211-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.657495 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656546 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.658331 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656577 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-socket-dir-parent\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.658331 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656632 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-run-k8s-cni-cncf-io\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.658331 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656658 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-daemon-config\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.658331 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656683 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j76cj\" (UniqueName: \"kubernetes.io/projected/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-kube-api-access-j76cj\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.658331 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656706 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-sysconfig\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.658331 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656764 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrvbt\" (UniqueName: \"kubernetes.io/projected/f35c9e60-a8e9-43b2-bea6-614f6555bc03-kube-api-access-vrvbt\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.658331 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656771 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-env-overrides\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.658331 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656787 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-run-systemd\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.658331 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656808 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-run-openvswitch\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.658331 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656826 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.658331 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656830 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-system-cni-dir\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.658331 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656865 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-run-k8s-cni-cncf-io\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.658331 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656882 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-system-cni-dir\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.658331 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.656908 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e7af4112-7294-4d61-95bd-d05625f88bdc-iptables-alerter-script\") pod \"iptables-alerter-4rj8r\" (UID: \"e7af4112-7294-4d61-95bd-d05625f88bdc\") " pod="openshift-network-operator/iptables-alerter-4rj8r" Oct 30 12:51:08.658331 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657097 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-sysconfig\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.658331 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657394 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-socket-dir-parent\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.658331 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657453 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs\") pod \"network-metrics-daemon-7l5g4\" (UID: \"6b553c14-1755-44ed-8dec-a2d1d8259c1d\") " pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:08.659204 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657486 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-sysctl-d\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.659204 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657511 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-var-lib-kubelet\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.659204 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657578 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndhbb\" (UniqueName: \"kubernetes.io/projected/10b4d740-fb6d-486e-b6c1-5c6115a10496-kube-api-access-ndhbb\") pod \"node-ca-kvsqc\" (UID: \"10b4d740-fb6d-486e-b6c1-5c6115a10496\") " pod="openshift-image-registry/node-ca-kvsqc" Oct 30 12:51:08.659204 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657609 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c10fc6a1-af69-48c2-b3dc-131e48f36b22-hosts-file\") pod \"node-resolver-z6m2l\" (UID: \"c10fc6a1-af69-48c2-b3dc-131e48f36b22\") " pod="openshift-dns/node-resolver-z6m2l" Oct 30 12:51:08.659204 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657635 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-cni-binary-copy\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.659204 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657652 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc2355fd-e26c-4d96-b399-5528863ba211-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.659204 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657662 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-run-netns\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.659204 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657689 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-etc-kubernetes\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.659204 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657722 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-run-openvswitch\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.659204 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657779 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-sys-fs\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.659204 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657789 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-daemon-config\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.659204 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657811 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-run-netns\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.659204 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657843 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-node-log\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.659204 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657858 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-var-lib-kubelet\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.659204 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657871 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-hostroot\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.659204 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657896 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-tuned\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.659204 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657898 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-run-systemd\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.659204 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657920 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-kubelet\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.660174 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:08.657930 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:08.660174 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657928 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-sysctl-d\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.660174 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657985 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-hostroot\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.660174 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.657990 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-sys-fs\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.660174 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.658036 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-run-netns\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.660174 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.658037 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-etc-kubernetes\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.660174 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.658065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct5kf\" (UniqueName: \"kubernetes.io/projected/fc2355fd-e26c-4d96-b399-5528863ba211-kube-api-access-ct5kf\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.660174 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.658076 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-node-log\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.660174 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.658092 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-host\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.660174 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.658306 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-host\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.660174 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.658348 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-kubelet\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.660174 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.658397 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c10fc6a1-af69-48c2-b3dc-131e48f36b22-hosts-file\") pod \"node-resolver-z6m2l\" (UID: \"c10fc6a1-af69-48c2-b3dc-131e48f36b22\") " pod="openshift-dns/node-resolver-z6m2l" Oct 30 12:51:08.660174 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.658432 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-run-netns\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.660174 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:08.658497 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs podName:6b553c14-1755-44ed-8dec-a2d1d8259c1d nodeName:}" failed. No retries permitted until 2026-10-30 12:51:09.158478851 +0000 UTC m=+3.058507909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs") pod "network-metrics-daemon-7l5g4" (UID: "6b553c14-1755-44ed-8dec-a2d1d8259c1d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:08.660174 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.658819 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-cni-binary-copy\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.660734 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.660611 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f35c9e60-a8e9-43b2-bea6-614f6555bc03-tmp\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.660790 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.660760 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-ovn-node-metrics-cert\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.660853 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.660769 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51f1ecdc-2207-4ac7-9193-77f917cc0143-agent-certs\") pod \"konnectivity-agent-pz66k\" (UID: \"51f1ecdc-2207-4ac7-9193-77f917cc0143\") " pod="kube-system/konnectivity-agent-pz66k" Oct 30 12:51:08.661070 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.661050 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-tuned\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.683855 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:08.683783 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:08.683855 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:08.683812 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:08.683855 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:08.683827 2581 projected.go:196] Error preparing data for projected volume kube-api-access-2lfnw for pod openshift-network-diagnostics/network-check-target-xb4w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:08.684146 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:08.683899 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw podName:30b22e49-8949-4dc1-9886-cc25b97f3436 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:09.183877746 +0000 UTC m=+3.083906805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2lfnw" (UniqueName: "kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw") pod "network-check-target-xb4w4" (UID: "30b22e49-8949-4dc1-9886-cc25b97f3436") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:08.686415 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.686043 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpmwj\" (UniqueName: \"kubernetes.io/projected/061dda5b-09a3-493d-9c16-6d25055d5b82-kube-api-access-qpmwj\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.686415 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.686043 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7gmw\" (UniqueName: \"kubernetes.io/projected/6b553c14-1755-44ed-8dec-a2d1d8259c1d-kube-api-access-f7gmw\") pod \"network-metrics-daemon-7l5g4\" (UID: \"6b553c14-1755-44ed-8dec-a2d1d8259c1d\") " pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:08.690775 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.690738 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tqpc\" (UniqueName: \"kubernetes.io/projected/e7af4112-7294-4d61-95bd-d05625f88bdc-kube-api-access-9tqpc\") pod \"iptables-alerter-4rj8r\" (UID: \"e7af4112-7294-4d61-95bd-d05625f88bdc\") " pod="openshift-network-operator/iptables-alerter-4rj8r" Oct 30 12:51:08.690905 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.690801 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6vzs\" (UniqueName: \"kubernetes.io/projected/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-kube-api-access-v6vzs\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.691492 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.691463 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49rmt\" (UniqueName: \"kubernetes.io/projected/c10fc6a1-af69-48c2-b3dc-131e48f36b22-kube-api-access-49rmt\") pod \"node-resolver-z6m2l\" (UID: \"c10fc6a1-af69-48c2-b3dc-131e48f36b22\") " pod="openshift-dns/node-resolver-z6m2l" Oct 30 12:51:08.695219 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.695101 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j76cj\" (UniqueName: \"kubernetes.io/projected/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-kube-api-access-j76cj\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.696799 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.696706 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct5kf\" (UniqueName: \"kubernetes.io/projected/fc2355fd-e26c-4d96-b399-5528863ba211-kube-api-access-ct5kf\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.697274 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.697252 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrvbt\" (UniqueName: \"kubernetes.io/projected/f35c9e60-a8e9-43b2-bea6-614f6555bc03-kube-api-access-vrvbt\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.697361 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.697297 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndhbb\" (UniqueName: \"kubernetes.io/projected/10b4d740-fb6d-486e-b6c1-5c6115a10496-kube-api-access-ndhbb\") pod \"node-ca-kvsqc\" (UID: \"10b4d740-fb6d-486e-b6c1-5c6115a10496\") " pod="openshift-image-registry/node-ca-kvsqc" Oct 30 12:51:08.745814 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.745775 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:08.785241 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.785210 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4rj8r" Oct 30 12:51:08.799855 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.799824 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:08.802819 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.802798 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z6m2l" Oct 30 12:51:08.804154 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:08.804120 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7af4112_7294_4d61_95bd_d05625f88bdc.slice/crio-6f9d5919fe698693d8dd529bcb2b5d881abb7d72aa714248cc325c884212522d WatchSource:0}: Error finding container 6f9d5919fe698693d8dd529bcb2b5d881abb7d72aa714248cc325c884212522d: Status 404 returned error can't find the container with id 6f9d5919fe698693d8dd529bcb2b5d881abb7d72aa714248cc325c884212522d Oct 30 12:51:08.808607 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.808575 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8l62c" Oct 30 12:51:08.811478 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:08.811443 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2355fd_e26c_4d96_b399_5528863ba211.slice/crio-cbab68617082441e7f1bef4668aec1d37f6d0dd7bf7596f51602340971802b93 WatchSource:0}: Error finding container cbab68617082441e7f1bef4668aec1d37f6d0dd7bf7596f51602340971802b93: Status 404 returned error can't find the container with id cbab68617082441e7f1bef4668aec1d37f6d0dd7bf7596f51602340971802b93 Oct 30 12:51:08.818467 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:08.818445 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9aa4593_872a_4501_9a7e_8cdeec5a92bf.slice/crio-2a889a230f994cb2ef99626cc18f96c135e3a0b2ede65a2a71b0d07f3b0fa098 WatchSource:0}: Error finding container 2a889a230f994cb2ef99626cc18f96c135e3a0b2ede65a2a71b0d07f3b0fa098: Status 404 returned error can't find the container with id 2a889a230f994cb2ef99626cc18f96c135e3a0b2ede65a2a71b0d07f3b0fa098 Oct 30 12:51:08.820513 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.820487 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:08.827312 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.827290 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:08.831037 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:08.831010 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061dda5b_09a3_493d_9c16_6d25055d5b82.slice/crio-4daea0f7cc943381817be60dcec8a8a9ae8e6ab8b33d6217107fc9094900e753 WatchSource:0}: Error finding container 4daea0f7cc943381817be60dcec8a8a9ae8e6ab8b33d6217107fc9094900e753: Status 404 returned error can't find the container with id 4daea0f7cc943381817be60dcec8a8a9ae8e6ab8b33d6217107fc9094900e753 Oct 30 12:51:08.836359 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.836195 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pz66k" Oct 30 12:51:08.836726 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:08.836675 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35c9e60_a8e9_43b2_bea6_614f6555bc03.slice/crio-d4544034ef1b796c44a3a012c4b460824464a9720a679358e9f9c15be0b5feeb WatchSource:0}: Error finding container d4544034ef1b796c44a3a012c4b460824464a9720a679358e9f9c15be0b5feeb: Status 404 returned error can't find the container with id d4544034ef1b796c44a3a012c4b460824464a9720a679358e9f9c15be0b5feeb Oct 30 12:51:08.844487 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.844463 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kvsqc" Oct 30 12:51:08.844872 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:08.844846 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f1ecdc_2207_4ac7_9193_77f917cc0143.slice/crio-de47d77f14791de8fa895481fe59f9832817859f428d48da0c3c39fcb5396a92 WatchSource:0}: Error finding container de47d77f14791de8fa895481fe59f9832817859f428d48da0c3c39fcb5396a92: Status 404 returned error can't find the container with id de47d77f14791de8fa895481fe59f9832817859f428d48da0c3c39fcb5396a92 Oct 30 12:51:08.856197 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:08.856172 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b4d740_fb6d_486e_b6c1_5c6115a10496.slice/crio-8acf9e7ab72f0281efb81256a559ac1c739935c6d5c72042bf94bfb50d8aecf5 WatchSource:0}: Error finding container 8acf9e7ab72f0281efb81256a559ac1c739935c6d5c72042bf94bfb50d8aecf5: Status 404 returned error can't find the container with id 8acf9e7ab72f0281efb81256a559ac1c739935c6d5c72042bf94bfb50d8aecf5 Oct 30 12:51:08.859900 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:08.859817 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:08.869777 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:08.869746 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc2a3e4_911a_4d26_bc60_65fc4b77cbb5.slice/crio-211fcceeb7cccbd0d573b4e75e74d6a9601b07a8b2d77d206e690a48baac2320 WatchSource:0}: Error finding container 211fcceeb7cccbd0d573b4e75e74d6a9601b07a8b2d77d206e690a48baac2320: Status 404 returned error can't find the container with id 211fcceeb7cccbd0d573b4e75e74d6a9601b07a8b2d77d206e690a48baac2320 Oct 30 12:51:09.161357 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:09.161329 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs\") pod \"network-metrics-daemon-7l5g4\" (UID: \"6b553c14-1755-44ed-8dec-a2d1d8259c1d\") " pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:09.161478 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:09.161370 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:09.161478 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:09.161469 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:09.161562 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:09.161525 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret podName:87f27309-51ca-4116-b0e9-10cf9e144c36 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:10.161512368 +0000 UTC m=+4.061541410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret") pod "global-pull-secret-syncer-r5lrn" (UID: "87f27309-51ca-4116-b0e9-10cf9e144c36") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:09.161562 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:09.161469 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:09.161634 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:09.161573 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs podName:6b553c14-1755-44ed-8dec-a2d1d8259c1d nodeName:}" failed. No retries permitted until 2026-10-30 12:51:10.161563838 +0000 UTC m=+4.061592897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs") pod "network-metrics-daemon-7l5g4" (UID: "6b553c14-1755-44ed-8dec-a2d1d8259c1d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:09.262675 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:09.262002 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfnw\" (UniqueName: \"kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw\") pod \"network-check-target-xb4w4\" (UID: \"30b22e49-8949-4dc1-9886-cc25b97f3436\") " pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:09.262675 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:09.262204 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:09.262675 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:09.262229 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:09.262675 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:09.262244 2581 projected.go:196] Error preparing data for projected volume kube-api-access-2lfnw for pod openshift-network-diagnostics/network-check-target-xb4w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:09.262675 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:09.262310 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw podName:30b22e49-8949-4dc1-9886-cc25b97f3436 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:10.262292371 +0000 UTC m=+4.162321419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2lfnw" (UniqueName: "kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw") pod "network-check-target-xb4w4" (UID: "30b22e49-8949-4dc1-9886-cc25b97f3436") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:09.536738 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:09.536660 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:46:08 +0000 UTC" deadline="2028-08-16 15:16:18.217960854 +0000 UTC" Oct 30 12:51:09.536738 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:09.536737 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15746h25m8.681228779s" Oct 30 12:51:09.578530 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:09.578377 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kvsqc" event={"ID":"10b4d740-fb6d-486e-b6c1-5c6115a10496","Type":"ContainerStarted","Data":"8acf9e7ab72f0281efb81256a559ac1c739935c6d5c72042bf94bfb50d8aecf5"} Oct 30 12:51:09.594163 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:09.594120 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" event={"ID":"061dda5b-09a3-493d-9c16-6d25055d5b82","Type":"ContainerStarted","Data":"4daea0f7cc943381817be60dcec8a8a9ae8e6ab8b33d6217107fc9094900e753"} Oct 30 12:51:09.597175 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:09.597134 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8l62c" event={"ID":"f9aa4593-872a-4501-9a7e-8cdeec5a92bf","Type":"ContainerStarted","Data":"2a889a230f994cb2ef99626cc18f96c135e3a0b2ede65a2a71b0d07f3b0fa098"} Oct 30 12:51:09.607571 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:09.607506 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z6m2l" event={"ID":"c10fc6a1-af69-48c2-b3dc-131e48f36b22","Type":"ContainerStarted","Data":"d6011e018859e7d10b70f4ea2d1750e0b8e068413d0e24e89b6e52e7cce50f11"} Oct 30 12:51:09.620929 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:09.620893 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4rj8r" event={"ID":"e7af4112-7294-4d61-95bd-d05625f88bdc","Type":"ContainerStarted","Data":"6f9d5919fe698693d8dd529bcb2b5d881abb7d72aa714248cc325c884212522d"} Oct 30 12:51:09.634971 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:09.632188 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal" event={"ID":"f1dd3ede18217038ba719ab8e2dbe78f","Type":"ContainerStarted","Data":"194151d7c5b174859a6ed93bfd485e919d7586e01020f339c8dbffb3587cdedc"} Oct 30 12:51:09.639938 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:09.639897 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" event={"ID":"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5","Type":"ContainerStarted","Data":"211fcceeb7cccbd0d573b4e75e74d6a9601b07a8b2d77d206e690a48baac2320"} Oct 30 12:51:09.651444 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:09.651411 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pz66k" event={"ID":"51f1ecdc-2207-4ac7-9193-77f917cc0143","Type":"ContainerStarted","Data":"de47d77f14791de8fa895481fe59f9832817859f428d48da0c3c39fcb5396a92"} Oct 30 12:51:09.655488 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:09.655453 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" event={"ID":"f35c9e60-a8e9-43b2-bea6-614f6555bc03","Type":"ContainerStarted","Data":"d4544034ef1b796c44a3a012c4b460824464a9720a679358e9f9c15be0b5feeb"} Oct 30 12:51:09.657428 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:09.657398 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cl4bq" event={"ID":"fc2355fd-e26c-4d96-b399-5528863ba211","Type":"ContainerStarted","Data":"cbab68617082441e7f1bef4668aec1d37f6d0dd7bf7596f51602340971802b93"} Oct 30 12:51:09.673272 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:09.673238 2581 generic.go:356] "Generic (PLEG): container finished" podID="ba36f1221f97bdfc5568490402fa51dc" containerID="75a3fe2043c62b86c18ffab41e87a824dc1f41da973c243ac0eee7b26d5e2400" exitCode=0 Oct 30 12:51:09.673408 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:09.673285 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" event={"ID":"ba36f1221f97bdfc5568490402fa51dc","Type":"ContainerDied","Data":"75a3fe2043c62b86c18ffab41e87a824dc1f41da973c243ac0eee7b26d5e2400"} Oct 30 12:51:09.704322 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:09.704290 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:09.723351 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:09.723287 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal" podStartSLOduration=1.723267007 podStartE2EDuration="1.723267007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:09.677349344 +0000 UTC m=+3.577378412" watchObservedRunningTime="2026-10-30 12:51:09.723267007 +0000 UTC m=+3.623296075" Oct 30 12:51:10.171295 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:10.170926 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs\") pod \"network-metrics-daemon-7l5g4\" (UID: \"6b553c14-1755-44ed-8dec-a2d1d8259c1d\") " pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:10.171295 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:10.171241 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:10.171522 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:10.171179 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:10.171522 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:10.171434 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs podName:6b553c14-1755-44ed-8dec-a2d1d8259c1d nodeName:}" failed. No retries permitted until 2026-10-30 12:51:12.17141632 +0000 UTC m=+6.071445379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs") pod "network-metrics-daemon-7l5g4" (UID: "6b553c14-1755-44ed-8dec-a2d1d8259c1d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:10.171858 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:10.171372 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:10.171977 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:10.171904 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret podName:87f27309-51ca-4116-b0e9-10cf9e144c36 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:12.171887228 +0000 UTC m=+6.071916270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret") pod "global-pull-secret-syncer-r5lrn" (UID: "87f27309-51ca-4116-b0e9-10cf9e144c36") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:10.272623 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:10.272587 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfnw\" (UniqueName: \"kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw\") pod \"network-check-target-xb4w4\" (UID: \"30b22e49-8949-4dc1-9886-cc25b97f3436\") " pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:10.272860 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:10.272840 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:10.272943 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:10.272868 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:10.272943 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:10.272881 2581 projected.go:196] Error preparing data for projected volume kube-api-access-2lfnw for pod openshift-network-diagnostics/network-check-target-xb4w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:10.273124 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:10.272942 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw podName:30b22e49-8949-4dc1-9886-cc25b97f3436 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:12.272924908 +0000 UTC m=+6.172953967 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2lfnw" (UniqueName: "kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw") pod "network-check-target-xb4w4" (UID: "30b22e49-8949-4dc1-9886-cc25b97f3436") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:10.537920 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:10.537763 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:46:08 +0000 UTC" deadline="2028-07-01 23:21:59.441045693 +0000 UTC" Oct 30 12:51:10.537920 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:10.537833 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14650h30m48.903217694s" Oct 30 12:51:10.566001 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:10.565069 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:10.566001 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:10.565204 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5lrn" podUID="87f27309-51ca-4116-b0e9-10cf9e144c36" Oct 30 12:51:10.566001 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:10.565743 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:10.566001 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:10.565862 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb4w4" podUID="30b22e49-8949-4dc1-9886-cc25b97f3436" Oct 30 12:51:10.566001 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:10.565956 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:10.566458 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:10.566086 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l5g4" podUID="6b553c14-1755-44ed-8dec-a2d1d8259c1d" Oct 30 12:51:10.692503 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:10.692425 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" event={"ID":"ba36f1221f97bdfc5568490402fa51dc","Type":"ContainerStarted","Data":"5e6e360e8e218637ae620654081571c16cd743d47fa475d84648ace53570f88f"} Oct 30 12:51:10.739781 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:10.739720 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" podStartSLOduration=2.739702067 podStartE2EDuration="2.739702067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:10.739421396 +0000 UTC m=+4.639450466" watchObservedRunningTime="2026-10-30 12:51:10.739702067 +0000 UTC m=+4.639731135" Oct 30 12:51:12.191159 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:12.191122 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:12.191604 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:12.191237 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs\") pod \"network-metrics-daemon-7l5g4\" (UID: \"6b553c14-1755-44ed-8dec-a2d1d8259c1d\") " pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:12.191604 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:12.191380 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:12.191604 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:12.191448 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs podName:6b553c14-1755-44ed-8dec-a2d1d8259c1d nodeName:}" failed. No retries permitted until 2026-10-30 12:51:16.191428848 +0000 UTC m=+10.091457908 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs") pod "network-metrics-daemon-7l5g4" (UID: "6b553c14-1755-44ed-8dec-a2d1d8259c1d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:12.191907 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:12.191888 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:12.191963 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:12.191950 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret podName:87f27309-51ca-4116-b0e9-10cf9e144c36 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:16.191933544 +0000 UTC m=+10.091962595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret") pod "global-pull-secret-syncer-r5lrn" (UID: "87f27309-51ca-4116-b0e9-10cf9e144c36") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:12.292452 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:12.292346 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfnw\" (UniqueName: \"kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw\") pod \"network-check-target-xb4w4\" (UID: \"30b22e49-8949-4dc1-9886-cc25b97f3436\") " pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:12.292633 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:12.292534 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:12.292633 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:12.292558 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:12.292633 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:12.292571 2581 projected.go:196] Error preparing data for projected volume kube-api-access-2lfnw for pod openshift-network-diagnostics/network-check-target-xb4w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:12.292633 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:12.292633 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw podName:30b22e49-8949-4dc1-9886-cc25b97f3436 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:16.292613387 +0000 UTC m=+10.192642446 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2lfnw" (UniqueName: "kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw") pod "network-check-target-xb4w4" (UID: "30b22e49-8949-4dc1-9886-cc25b97f3436") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:12.564949 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:12.564609 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:12.564949 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:12.564747 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5lrn" podUID="87f27309-51ca-4116-b0e9-10cf9e144c36" Oct 30 12:51:12.564949 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:12.564810 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:12.564949 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:12.564877 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb4w4" podUID="30b22e49-8949-4dc1-9886-cc25b97f3436" Oct 30 12:51:12.564949 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:12.564923 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:12.565305 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:12.564995 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l5g4" podUID="6b553c14-1755-44ed-8dec-a2d1d8259c1d" Oct 30 12:51:12.714879 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:12.714843 2581 generic.go:356] "Generic (PLEG): container finished" podID="fc2355fd-e26c-4d96-b399-5528863ba211" containerID="677a730fe90161fd5e7af9630de35d05c50f5bcf96de08ea490447a8787e7cca" exitCode=0 Oct 30 12:51:12.715037 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:12.714899 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cl4bq" event={"ID":"fc2355fd-e26c-4d96-b399-5528863ba211","Type":"ContainerDied","Data":"677a730fe90161fd5e7af9630de35d05c50f5bcf96de08ea490447a8787e7cca"} Oct 30 12:51:13.725108 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:13.724328 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pz66k" event={"ID":"51f1ecdc-2207-4ac7-9193-77f917cc0143","Type":"ContainerStarted","Data":"3b8b8f69cdb0ebf950810e8ff33a1fe6327b7b7650cdf96096806fe6461578b2"} Oct 30 12:51:14.564428 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:14.564055 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:14.564428 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:14.564182 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l5g4" podUID="6b553c14-1755-44ed-8dec-a2d1d8259c1d" Oct 30 12:51:14.564428 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:14.564246 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:14.564428 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:14.564316 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5lrn" podUID="87f27309-51ca-4116-b0e9-10cf9e144c36" Oct 30 12:51:14.564721 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:14.564465 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:14.564721 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:14.564543 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb4w4" podUID="30b22e49-8949-4dc1-9886-cc25b97f3436" Oct 30 12:51:14.729489 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:14.729447 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kvsqc" event={"ID":"10b4d740-fb6d-486e-b6c1-5c6115a10496","Type":"ContainerStarted","Data":"39d36cb584367e567ec89c0dabe1b2fd51fb71b2d55c6a36a2776aa6e34249ca"} Oct 30 12:51:14.732963 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:14.732918 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" event={"ID":"061dda5b-09a3-493d-9c16-6d25055d5b82","Type":"ContainerStarted","Data":"8462e384a53808130e20269d3b69e06b6460c08f7952cbc9c96958d1ea3a6abc"} Oct 30 12:51:14.750449 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:14.750241 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pz66k" Oct 30 12:51:14.750981 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:14.750873 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pz66k" Oct 30 12:51:14.773978 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:14.773631 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pz66k" podStartSLOduration=4.099774461 podStartE2EDuration="7.773619192s" totalImagesPullingTime="3.673844731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:13.768281077 +0000 UTC m=+7.668310141" watchObservedRunningTime="2026-10-30 12:51:14.773619192 +0000 UTC m=+8.673648252" Oct 30 12:51:14.773978 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:14.773722 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kvsqc" podStartSLOduration=2.666209757 podStartE2EDuration="7.773716842s" totalImagesPullingTime="5.107507085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:14.773141513 +0000 UTC m=+8.673170577" watchObservedRunningTime="2026-10-30 12:51:14.773716842 +0000 UTC m=+8.673745906" Oct 30 12:51:16.061921 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:16.061895 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 12:51:16.232985 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:16.232154 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs\") pod \"network-metrics-daemon-7l5g4\" (UID: \"6b553c14-1755-44ed-8dec-a2d1d8259c1d\") " pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:16.232985 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:16.232212 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:16.232985 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:16.232335 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:16.232985 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:16.232395 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret podName:87f27309-51ca-4116-b0e9-10cf9e144c36 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:24.232378579 +0000 UTC m=+18.132407636 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret") pod "global-pull-secret-syncer-r5lrn" (UID: "87f27309-51ca-4116-b0e9-10cf9e144c36") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:16.232985 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:16.232811 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:16.232985 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:16.232871 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs podName:6b553c14-1755-44ed-8dec-a2d1d8259c1d nodeName:}" failed. No retries permitted until 2026-10-30 12:51:24.232853256 +0000 UTC m=+18.132882432 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs") pod "network-metrics-daemon-7l5g4" (UID: "6b553c14-1755-44ed-8dec-a2d1d8259c1d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:16.333188 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:16.333147 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfnw\" (UniqueName: \"kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw\") pod \"network-check-target-xb4w4\" (UID: \"30b22e49-8949-4dc1-9886-cc25b97f3436\") " pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:16.333365 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:16.333338 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:16.333365 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:16.333362 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:16.333477 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:16.333375 2581 projected.go:196] Error preparing data for projected volume kube-api-access-2lfnw for pod openshift-network-diagnostics/network-check-target-xb4w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:16.333477 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:16.333461 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw podName:30b22e49-8949-4dc1-9886-cc25b97f3436 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:24.333426503 +0000 UTC m=+18.233455558 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2lfnw" (UniqueName: "kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw") pod "network-check-target-xb4w4" (UID: "30b22e49-8949-4dc1-9886-cc25b97f3436") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:16.492439 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:16.491586 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:51:16.061919437Z","UUID":"217ecc6e-9b95-437d-8c7f-734cdb1d0b55","Handler":null,"Name":"","Endpoint":""} Oct 30 12:51:16.495493 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:16.494712 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 12:51:16.495493 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:16.494750 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 12:51:16.563815 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:16.563781 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:16.563970 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:16.563904 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5lrn" podUID="87f27309-51ca-4116-b0e9-10cf9e144c36" Oct 30 12:51:16.564050 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:16.563994 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:16.564107 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:16.564062 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb4w4" podUID="30b22e49-8949-4dc1-9886-cc25b97f3436" Oct 30 12:51:16.564262 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:16.564221 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:16.564465 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:16.564313 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l5g4" podUID="6b553c14-1755-44ed-8dec-a2d1d8259c1d" Oct 30 12:51:16.740764 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:16.740061 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8l62c" event={"ID":"f9aa4593-872a-4501-9a7e-8cdeec5a92bf","Type":"ContainerStarted","Data":"139e97df06fbd769e423e8c5624483ef3b21b5d9c69002d7d5147c7cf64c401f"} Oct 30 12:51:16.745793 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:16.745246 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 12:51:16.745793 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:16.745694 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" event={"ID":"061dda5b-09a3-493d-9c16-6d25055d5b82","Type":"ContainerStarted","Data":"beba6eabec2f9daf13ed38c5bb1b9fcc2ba4348e023dcbfb8a192ba16f008395"} Oct 30 12:51:16.785538 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:16.785492 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-8l62c" podStartSLOduration=2.147278277 podStartE2EDuration="9.785479495s" totalImagesPullingTime="7.638201218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:16.785373411 +0000 UTC m=+10.685402473" watchObservedRunningTime="2026-10-30 12:51:16.785479495 +0000 UTC m=+10.685508558" Oct 30 12:51:17.751808 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:17.751768 2581 generic.go:356] "Generic (PLEG): container finished" podID="fc2355fd-e26c-4d96-b399-5528863ba211" containerID="22bfdd65ed8537f2640e76c783e7309025712cb82d315ffd4bb742f485587e89" exitCode=0 Oct 30 12:51:17.752209 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:17.751874 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cl4bq" event={"ID":"fc2355fd-e26c-4d96-b399-5528863ba211","Type":"ContainerDied","Data":"22bfdd65ed8537f2640e76c783e7309025712cb82d315ffd4bb742f485587e89"} Oct 30 12:51:17.756456 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:17.756434 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" event={"ID":"061dda5b-09a3-493d-9c16-6d25055d5b82","Type":"ContainerStarted","Data":"79eb7f19ce0694803e3e96b3ed6ca451074da8aa93b788355fea8a016c1b59ba"} Oct 30 12:51:17.758488 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:17.758462 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z6m2l" event={"ID":"c10fc6a1-af69-48c2-b3dc-131e48f36b22","Type":"ContainerStarted","Data":"c08290e125b9fc6a2b6c633bd12c66d28ff8eca9398ddc1c457d8b5103abf4e9"} Oct 30 12:51:17.845892 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:17.845810 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" podStartSLOduration=2.829771192 podStartE2EDuration="10.84579387s" totalImagesPullingTime="8.016022678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:17.845667824 +0000 UTC m=+11.745696887" watchObservedRunningTime="2026-10-30 12:51:17.84579387 +0000 UTC m=+11.745822936" Oct 30 12:51:17.888102 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:17.888036 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-z6m2l" podStartSLOduration=2.656788662 podStartE2EDuration="10.888016818s" totalImagesPullingTime="8.231228156s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:17.887427685 +0000 UTC m=+11.787456749" watchObservedRunningTime="2026-10-30 12:51:17.888016818 +0000 UTC m=+11.788045884" Oct 30 12:51:18.563242 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:18.563043 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:18.564558 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:18.563125 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:18.564558 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:18.563333 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb4w4" podUID="30b22e49-8949-4dc1-9886-cc25b97f3436" Oct 30 12:51:18.564558 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:18.563400 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5lrn" podUID="87f27309-51ca-4116-b0e9-10cf9e144c36" Oct 30 12:51:18.564558 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:18.563198 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:18.564558 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:18.563548 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l5g4" podUID="6b553c14-1755-44ed-8dec-a2d1d8259c1d" Oct 30 12:51:18.761477 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:18.761437 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4rj8r" event={"ID":"e7af4112-7294-4d61-95bd-d05625f88bdc","Type":"ContainerStarted","Data":"e49b55e5e795dd976c7fcc2ee2c4b5649af7260dfc8beceed18ba2798205cf17"} Oct 30 12:51:18.803774 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:18.803714 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4rj8r" podStartSLOduration=3.8788608460000003 podStartE2EDuration="11.803693976s" totalImagesPullingTime="7.92483313s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:18.80339804 +0000 UTC m=+12.703427103" watchObservedRunningTime="2026-10-30 12:51:18.803693976 +0000 UTC m=+12.703723041" Oct 30 12:51:19.766725 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:19.766688 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" event={"ID":"f35c9e60-a8e9-43b2-bea6-614f6555bc03","Type":"ContainerStarted","Data":"b3a8a8c1d95e135212e011213d9b19161c09337e816424e5e565d666e05ad3a4"} Oct 30 12:51:19.769851 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:19.769824 2581 generic.go:356] "Generic (PLEG): container finished" podID="fc2355fd-e26c-4d96-b399-5528863ba211" containerID="49fc33c41e70a7ff666b4f86f335be08a970cfdc41556106cd071bb3d6b4bb3d" exitCode=0 Oct 30 12:51:19.769994 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:19.769912 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cl4bq" event={"ID":"fc2355fd-e26c-4d96-b399-5528863ba211","Type":"ContainerDied","Data":"49fc33c41e70a7ff666b4f86f335be08a970cfdc41556106cd071bb3d6b4bb3d"} Oct 30 12:51:19.817887 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:19.817827 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" podStartSLOduration=2.489923515 podStartE2EDuration="12.817810229s" totalImagesPullingTime="10.327886714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:19.817489188 +0000 UTC m=+13.717518260" watchObservedRunningTime="2026-10-30 12:51:19.817810229 +0000 UTC m=+13.717839292" Oct 30 12:51:20.569044 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:20.566182 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:20.569044 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:20.566361 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l5g4" podUID="6b553c14-1755-44ed-8dec-a2d1d8259c1d" Oct 30 12:51:20.569044 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:20.566453 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:20.569044 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:20.566545 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5lrn" podUID="87f27309-51ca-4116-b0e9-10cf9e144c36" Oct 30 12:51:20.569044 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:20.566609 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:20.569044 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:20.566715 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb4w4" podUID="30b22e49-8949-4dc1-9886-cc25b97f3436" Oct 30 12:51:20.775213 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:20.775174 2581 generic.go:356] "Generic (PLEG): container finished" podID="fc2355fd-e26c-4d96-b399-5528863ba211" containerID="58f583c5ba8c93eb6047fd0f6ff15a27961482021a988dac5396945f454163c9" exitCode=0 Oct 30 12:51:20.775806 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:20.775222 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cl4bq" event={"ID":"fc2355fd-e26c-4d96-b399-5528863ba211","Type":"ContainerDied","Data":"58f583c5ba8c93eb6047fd0f6ff15a27961482021a988dac5396945f454163c9"} Oct 30 12:51:22.562659 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:22.562446 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:22.563349 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:22.562446 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:22.563349 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:22.562774 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l5g4" podUID="6b553c14-1755-44ed-8dec-a2d1d8259c1d" Oct 30 12:51:22.563349 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:22.562471 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:22.563349 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:22.562825 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5lrn" podUID="87f27309-51ca-4116-b0e9-10cf9e144c36" Oct 30 12:51:22.563349 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:22.562929 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb4w4" podUID="30b22e49-8949-4dc1-9886-cc25b97f3436" Oct 30 12:51:22.781527 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:22.781485 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cl4bq" event={"ID":"fc2355fd-e26c-4d96-b399-5528863ba211","Type":"ContainerStarted","Data":"fbe92d0f15191521b9f2642ac5822b0f80c2cf2174dc4cd0dfd961dd3f2bbd06"} Oct 30 12:51:23.785801 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:23.785768 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" event={"ID":"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5","Type":"ContainerStarted","Data":"ad34ed1c11073fcfba6ebfe8042cbfa3e1b2473a649a875caf1de2a976b45f13"} Oct 30 12:51:23.786165 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:23.785892 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" event={"ID":"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5","Type":"ContainerStarted","Data":"62148b1c79c1da5da560b2e82ba57dc2a5c9369f565bcb18f6c1fc1c437cab18"} Oct 30 12:51:23.786165 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:23.785972 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" event={"ID":"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5","Type":"ContainerStarted","Data":"9d93b1671452dff881f0b2f0e98e3d529e6f96760a7e7666be702a9543f32e88"} Oct 30 12:51:23.786165 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:23.786007 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" event={"ID":"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5","Type":"ContainerStarted","Data":"ca1c7c6addb62f7b89f0f2266d84e1f77def3cdd6d5109b6c8ddf39b5a4baaea"} Oct 30 12:51:23.788599 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:23.788574 2581 generic.go:356] "Generic (PLEG): container finished" podID="fc2355fd-e26c-4d96-b399-5528863ba211" containerID="fbe92d0f15191521b9f2642ac5822b0f80c2cf2174dc4cd0dfd961dd3f2bbd06" exitCode=0 Oct 30 12:51:23.788680 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:23.788610 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cl4bq" event={"ID":"fc2355fd-e26c-4d96-b399-5528863ba211","Type":"ContainerDied","Data":"fbe92d0f15191521b9f2642ac5822b0f80c2cf2174dc4cd0dfd961dd3f2bbd06"} Oct 30 12:51:24.285840 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:24.285798 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs\") pod \"network-metrics-daemon-7l5g4\" (UID: \"6b553c14-1755-44ed-8dec-a2d1d8259c1d\") " pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:24.285840 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:24.285847 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:24.286104 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:24.285959 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:24.286104 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:24.285966 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:24.286104 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:24.286009 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret podName:87f27309-51ca-4116-b0e9-10cf9e144c36 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:40.285998034 +0000 UTC m=+34.186027075 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret") pod "global-pull-secret-syncer-r5lrn" (UID: "87f27309-51ca-4116-b0e9-10cf9e144c36") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:24.286104 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:24.286031 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs podName:6b553c14-1755-44ed-8dec-a2d1d8259c1d nodeName:}" failed. No retries permitted until 2026-10-30 12:51:40.286016172 +0000 UTC m=+34.186045218 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs") pod "network-metrics-daemon-7l5g4" (UID: "6b553c14-1755-44ed-8dec-a2d1d8259c1d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:24.386468 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:24.386428 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfnw\" (UniqueName: \"kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw\") pod \"network-check-target-xb4w4\" (UID: \"30b22e49-8949-4dc1-9886-cc25b97f3436\") " pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:24.386627 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:24.386593 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:24.386627 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:24.386614 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:24.386627 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:24.386624 2581 projected.go:196] Error preparing data for projected volume kube-api-access-2lfnw for pod openshift-network-diagnostics/network-check-target-xb4w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:24.386722 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:24.386675 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw podName:30b22e49-8949-4dc1-9886-cc25b97f3436 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:40.386661344 +0000 UTC m=+34.286690399 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2lfnw" (UniqueName: "kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw") pod "network-check-target-xb4w4" (UID: "30b22e49-8949-4dc1-9886-cc25b97f3436") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:24.562959 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:24.562855 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:24.562959 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:24.562915 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:24.562959 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:24.562927 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:24.563170 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:24.563048 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb4w4" podUID="30b22e49-8949-4dc1-9886-cc25b97f3436" Oct 30 12:51:24.563203 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:24.563161 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5lrn" podUID="87f27309-51ca-4116-b0e9-10cf9e144c36" Oct 30 12:51:24.563253 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:24.563238 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l5g4" podUID="6b553c14-1755-44ed-8dec-a2d1d8259c1d" Oct 30 12:51:24.793187 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:24.793153 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" event={"ID":"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5","Type":"ContainerStarted","Data":"a6c3ed800427e3781188db076eb16c75ad9dd631d5fb53df19a627dee1e8af3e"} Oct 30 12:51:24.793708 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:24.793224 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" event={"ID":"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5","Type":"ContainerStarted","Data":"e79aa9cef248ed5550b6950c592ebec1f6ba0e4a44aa9dee7cad38180f6a0610"} Oct 30 12:51:24.795974 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:24.795949 2581 generic.go:356] "Generic (PLEG): container finished" podID="fc2355fd-e26c-4d96-b399-5528863ba211" containerID="525246909e3c978f648cd09eda8fa7b9553615436497eb6816bacb3695084d87" exitCode=0 Oct 30 12:51:24.796062 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:24.795981 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cl4bq" event={"ID":"fc2355fd-e26c-4d96-b399-5528863ba211","Type":"ContainerDied","Data":"525246909e3c978f648cd09eda8fa7b9553615436497eb6816bacb3695084d87"} Oct 30 12:51:25.035460 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:25.035422 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pz66k" Oct 30 12:51:25.035662 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:25.035558 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 12:51:25.036016 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:25.036000 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pz66k" Oct 30 12:51:25.801456 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:25.801417 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cl4bq" event={"ID":"fc2355fd-e26c-4d96-b399-5528863ba211","Type":"ContainerStarted","Data":"9fe0cc9cc23b800f7bfd4ec0cfb02e2400284bac1b2b18fe31c712e8db4cced9"} Oct 30 12:51:25.876118 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:25.876067 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cl4bq" podStartSLOduration=7.44332514 podStartE2EDuration="18.876051748s" totalImagesPullingTime="11.432726608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:25.863366306 +0000 UTC m=+19.763395387" watchObservedRunningTime="2026-10-30 12:51:25.876051748 +0000 UTC m=+19.776080811" Oct 30 12:51:26.563593 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:26.563560 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:26.563766 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:26.563706 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:26.563766 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:26.563717 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb4w4" podUID="30b22e49-8949-4dc1-9886-cc25b97f3436" Oct 30 12:51:26.563868 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:26.563830 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l5g4" podUID="6b553c14-1755-44ed-8dec-a2d1d8259c1d" Oct 30 12:51:26.563964 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:26.563892 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:26.564013 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:26.563988 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5lrn" podUID="87f27309-51ca-4116-b0e9-10cf9e144c36" Oct 30 12:51:26.807144 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:26.807104 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" event={"ID":"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5","Type":"ContainerStarted","Data":"d16d28c72a87991fcb09849b7e5d41e07fb23cdf946eb872891ae96b1539e8d9"} Oct 30 12:51:28.563209 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:28.563172 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:28.563209 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:28.563199 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:28.563756 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:28.563328 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb4w4" podUID="30b22e49-8949-4dc1-9886-cc25b97f3436" Oct 30 12:51:28.563756 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:28.563354 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:28.563756 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:28.563447 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5lrn" podUID="87f27309-51ca-4116-b0e9-10cf9e144c36" Oct 30 12:51:28.563756 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:28.563533 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l5g4" podUID="6b553c14-1755-44ed-8dec-a2d1d8259c1d" Oct 30 12:51:28.813873 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:28.813782 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" event={"ID":"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5","Type":"ContainerStarted","Data":"f4c4dd5749164a25336b37798b1a8c5ef0880890e2aed0426b0b0445b906ff51"} Oct 30 12:51:28.814106 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:28.814084 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:28.814190 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:28.814157 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:28.830405 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:28.830373 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:28.906557 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:28.906508 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" podStartSLOduration=7.274109489 podStartE2EDuration="21.906496277s" totalImagesPullingTime="14.632386788s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:28.906319909 +0000 UTC m=+22.806348972" watchObservedRunningTime="2026-10-30 12:51:28.906496277 +0000 UTC m=+22.806525340" Oct 30 12:51:29.816596 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:29.816569 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:29.831224 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:29.831197 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:30.135062 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:30.134992 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r5lrn"] Oct 30 12:51:30.135218 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:30.135131 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:30.135256 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:30.135220 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5lrn" podUID="87f27309-51ca-4116-b0e9-10cf9e144c36" Oct 30 12:51:30.137842 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:30.137798 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xb4w4"] Oct 30 12:51:30.138011 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:30.137925 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:30.138075 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:30.138041 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb4w4" podUID="30b22e49-8949-4dc1-9886-cc25b97f3436" Oct 30 12:51:30.139097 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:30.139073 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7l5g4"] Oct 30 12:51:30.139203 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:30.139160 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:30.139271 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:30.139227 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l5g4" podUID="6b553c14-1755-44ed-8dec-a2d1d8259c1d" Oct 30 12:51:31.562472 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:31.562431 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:31.563104 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:31.562431 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:31.563104 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:31.562562 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:31.563104 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:31.562564 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5lrn" podUID="87f27309-51ca-4116-b0e9-10cf9e144c36" Oct 30 12:51:31.563104 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:31.562631 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb4w4" podUID="30b22e49-8949-4dc1-9886-cc25b97f3436" Oct 30 12:51:31.563104 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:31.562691 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l5g4" podUID="6b553c14-1755-44ed-8dec-a2d1d8259c1d" Oct 30 12:51:33.563101 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:33.563068 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:33.563611 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:33.563113 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:33.563611 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:33.563166 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:33.563611 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:33.563179 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l5g4" podUID="6b553c14-1755-44ed-8dec-a2d1d8259c1d" Oct 30 12:51:33.563611 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:33.563232 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb4w4" podUID="30b22e49-8949-4dc1-9886-cc25b97f3436" Oct 30 12:51:33.563611 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:33.563321 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5lrn" podUID="87f27309-51ca-4116-b0e9-10cf9e144c36" Oct 30 12:51:35.562690 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:35.562583 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:35.562690 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:35.562615 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:35.563242 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:35.562714 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:35.563242 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:35.562732 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l5g4" podUID="6b553c14-1755-44ed-8dec-a2d1d8259c1d" Oct 30 12:51:35.563242 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:35.562789 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5lrn" podUID="87f27309-51ca-4116-b0e9-10cf9e144c36" Oct 30 12:51:35.563242 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:35.562834 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xb4w4" podUID="30b22e49-8949-4dc1-9886-cc25b97f3436" Oct 30 12:51:36.401005 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.400914 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeReady" Oct 30 12:51:36.401271 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.401087 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 12:51:36.468401 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.468367 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-p8vd8"] Oct 30 12:51:36.479474 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.479438 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b7kx7"] Oct 30 12:51:36.479647 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.479576 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p8vd8" Oct 30 12:51:36.481976 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.481954 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:36.492242 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.492216 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hbsfb"] Oct 30 12:51:36.495169 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.495145 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:36.496766 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.496146 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p8vd8"] Oct 30 12:51:36.498152 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.498129 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 12:51:36.498277 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.498261 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bhhxq\"" Oct 30 12:51:36.507968 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.507948 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 12:51:36.507968 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.507961 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 12:51:36.509058 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.509043 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 12:51:36.522546 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.522518 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hbsfb"] Oct 30 12:51:36.525298 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.525268 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b7kx7"] Oct 30 12:51:36.549075 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.548861 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 12:51:36.549075 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.549067 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ktwvb\"" Oct 30 12:51:36.549255 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.549069 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 12:51:36.549255 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.549129 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-45hdr\"" Oct 30 12:51:36.563679 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.563640 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zskvb\" (UniqueName: \"kubernetes.io/projected/6e7c1377-fb2f-4b5f-83a7-b799206fb344-kube-api-access-zskvb\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:36.563679 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.563670 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f2e503d-c234-415b-bce5-ec542e09d50b-cert\") pod \"ingress-canary-p8vd8\" (UID: \"1f2e503d-c234-415b-bce5-ec542e09d50b\") " pod="openshift-ingress-canary/ingress-canary-p8vd8" Oct 30 12:51:36.564179 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.563699 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e7c1377-fb2f-4b5f-83a7-b799206fb344-metrics-tls\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:36.564179 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.563810 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6crr\" (UniqueName: \"kubernetes.io/projected/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-api-access-f6crr\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:36.564179 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.563840 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7aad1c89-4805-4c51-a9dc-26db67c46f46-crio-socket\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:36.564179 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.563855 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7aad1c89-4805-4c51-a9dc-26db67c46f46-data-volume\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:36.564179 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.563927 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:36.564179 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.564021 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:36.564179 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.564055 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7c1377-fb2f-4b5f-83a7-b799206fb344-config-volume\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:36.564179 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.564076 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e7c1377-fb2f-4b5f-83a7-b799206fb344-tmp-dir\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:36.564179 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.564099 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bldsp\" (UniqueName: \"kubernetes.io/projected/1f2e503d-c234-415b-bce5-ec542e09d50b-kube-api-access-bldsp\") pod \"ingress-canary-p8vd8\" (UID: \"1f2e503d-c234-415b-bce5-ec542e09d50b\") " pod="openshift-ingress-canary/ingress-canary-p8vd8" Oct 30 12:51:36.613640 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.613605 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 12:51:36.613799 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.613739 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 12:51:36.615219 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.615201 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 12:51:36.664976 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.664894 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:36.664976 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.664930 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7c1377-fb2f-4b5f-83a7-b799206fb344-config-volume\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:36.664976 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.664973 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e7c1377-fb2f-4b5f-83a7-b799206fb344-tmp-dir\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:36.665175 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.664996 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bldsp\" (UniqueName: \"kubernetes.io/projected/1f2e503d-c234-415b-bce5-ec542e09d50b-kube-api-access-bldsp\") pod \"ingress-canary-p8vd8\" (UID: \"1f2e503d-c234-415b-bce5-ec542e09d50b\") " pod="openshift-ingress-canary/ingress-canary-p8vd8" Oct 30 12:51:36.665175 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:36.665022 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 12:51:36.665175 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.665031 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zskvb\" (UniqueName: \"kubernetes.io/projected/6e7c1377-fb2f-4b5f-83a7-b799206fb344-kube-api-access-zskvb\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:36.665175 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:36.665105 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-rbac-proxy-cm podName:7aad1c89-4805-4c51-a9dc-26db67c46f46 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:37.165084164 +0000 UTC m=+31.065113227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hbsfb" (UID: "7aad1c89-4805-4c51-a9dc-26db67c46f46") : configmap "kube-rbac-proxy" not found Oct 30 12:51:36.665175 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.665154 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f2e503d-c234-415b-bce5-ec542e09d50b-cert\") pod \"ingress-canary-p8vd8\" (UID: \"1f2e503d-c234-415b-bce5-ec542e09d50b\") " pod="openshift-ingress-canary/ingress-canary-p8vd8" Oct 30 12:51:36.665363 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.665198 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e7c1377-fb2f-4b5f-83a7-b799206fb344-metrics-tls\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:36.665363 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.665232 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6crr\" (UniqueName: \"kubernetes.io/projected/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-api-access-f6crr\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:36.665363 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.665253 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7aad1c89-4805-4c51-a9dc-26db67c46f46-crio-socket\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:36.665505 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:36.665380 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:51:36.665505 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.665445 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7aad1c89-4805-4c51-a9dc-26db67c46f46-data-volume\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:36.665505 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.665477 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7aad1c89-4805-4c51-a9dc-26db67c46f46-crio-socket\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:36.665505 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:36.665487 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e7c1377-fb2f-4b5f-83a7-b799206fb344-metrics-tls podName:6e7c1377-fb2f-4b5f-83a7-b799206fb344 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:37.165468824 +0000 UTC m=+31.065497882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6e7c1377-fb2f-4b5f-83a7-b799206fb344-metrics-tls") pod "dns-default-b7kx7" (UID: "6e7c1377-fb2f-4b5f-83a7-b799206fb344") : secret "dns-default-metrics-tls" not found Oct 30 12:51:36.665681 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.665569 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:36.665681 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:36.665630 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:51:36.665759 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:36.665687 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls podName:7aad1c89-4805-4c51-a9dc-26db67c46f46 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:37.165675562 +0000 UTC m=+31.065704603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbsfb" (UID: "7aad1c89-4805-4c51-a9dc-26db67c46f46") : secret "insights-runtime-extractor-tls" not found Oct 30 12:51:36.665879 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.665858 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7aad1c89-4805-4c51-a9dc-26db67c46f46-data-volume\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:36.665912 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.665876 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e7c1377-fb2f-4b5f-83a7-b799206fb344-tmp-dir\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:36.666122 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.666099 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7c1377-fb2f-4b5f-83a7-b799206fb344-config-volume\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:36.669219 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.669201 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f2e503d-c234-415b-bce5-ec542e09d50b-cert\") pod \"ingress-canary-p8vd8\" (UID: \"1f2e503d-c234-415b-bce5-ec542e09d50b\") " pod="openshift-ingress-canary/ingress-canary-p8vd8" Oct 30 12:51:36.697630 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.697600 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bldsp\" (UniqueName: \"kubernetes.io/projected/1f2e503d-c234-415b-bce5-ec542e09d50b-kube-api-access-bldsp\") pod \"ingress-canary-p8vd8\" (UID: \"1f2e503d-c234-415b-bce5-ec542e09d50b\") " pod="openshift-ingress-canary/ingress-canary-p8vd8" Oct 30 12:51:36.699769 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.699736 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6crr\" (UniqueName: \"kubernetes.io/projected/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-api-access-f6crr\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:36.700138 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.700118 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zskvb\" (UniqueName: \"kubernetes.io/projected/6e7c1377-fb2f-4b5f-83a7-b799206fb344-kube-api-access-zskvb\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:36.793118 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.793075 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p8vd8" Oct 30 12:51:36.832205 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.832164 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-f4rcm"] Oct 30 12:51:36.857322 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.857284 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-f4rcm"] Oct 30 12:51:36.857469 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.857406 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" Oct 30 12:51:36.881847 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.881800 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 12:51:36.881847 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.881812 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 12:51:36.881847 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.881837 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 12:51:36.882116 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.881877 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 12:51:36.882116 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.881898 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ksd25\"" Oct 30 12:51:36.967317 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.967282 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d4f771a8-7a67-429d-8701-9c3ffa29550c-signing-key\") pod \"service-ca-5fffc97bdf-f4rcm\" (UID: \"d4f771a8-7a67-429d-8701-9c3ffa29550c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" Oct 30 12:51:36.967491 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.967349 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzx7s\" (UniqueName: \"kubernetes.io/projected/d4f771a8-7a67-429d-8701-9c3ffa29550c-kube-api-access-jzx7s\") pod \"service-ca-5fffc97bdf-f4rcm\" (UID: \"d4f771a8-7a67-429d-8701-9c3ffa29550c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" Oct 30 12:51:36.967491 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:36.967408 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d4f771a8-7a67-429d-8701-9c3ffa29550c-signing-cabundle\") pod \"service-ca-5fffc97bdf-f4rcm\" (UID: \"d4f771a8-7a67-429d-8701-9c3ffa29550c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" Oct 30 12:51:37.068709 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.068673 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d4f771a8-7a67-429d-8701-9c3ffa29550c-signing-cabundle\") pod \"service-ca-5fffc97bdf-f4rcm\" (UID: \"d4f771a8-7a67-429d-8701-9c3ffa29550c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" Oct 30 12:51:37.068870 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.068753 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d4f771a8-7a67-429d-8701-9c3ffa29550c-signing-key\") pod \"service-ca-5fffc97bdf-f4rcm\" (UID: \"d4f771a8-7a67-429d-8701-9c3ffa29550c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" Oct 30 12:51:37.068870 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.068802 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzx7s\" (UniqueName: \"kubernetes.io/projected/d4f771a8-7a67-429d-8701-9c3ffa29550c-kube-api-access-jzx7s\") pod \"service-ca-5fffc97bdf-f4rcm\" (UID: \"d4f771a8-7a67-429d-8701-9c3ffa29550c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" Oct 30 12:51:37.069494 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.069464 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d4f771a8-7a67-429d-8701-9c3ffa29550c-signing-cabundle\") pod \"service-ca-5fffc97bdf-f4rcm\" (UID: \"d4f771a8-7a67-429d-8701-9c3ffa29550c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" Oct 30 12:51:37.071174 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.071157 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d4f771a8-7a67-429d-8701-9c3ffa29550c-signing-key\") pod \"service-ca-5fffc97bdf-f4rcm\" (UID: \"d4f771a8-7a67-429d-8701-9c3ffa29550c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" Oct 30 12:51:37.095935 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.095907 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzx7s\" (UniqueName: \"kubernetes.io/projected/d4f771a8-7a67-429d-8701-9c3ffa29550c-kube-api-access-jzx7s\") pod \"service-ca-5fffc97bdf-f4rcm\" (UID: \"d4f771a8-7a67-429d-8701-9c3ffa29550c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" Oct 30 12:51:37.169290 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.169252 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:37.169290 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.169292 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:37.169491 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:37.169405 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:51:37.169491 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:37.169427 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 12:51:37.169491 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:37.169470 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls podName:7aad1c89-4805-4c51-a9dc-26db67c46f46 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:38.169451652 +0000 UTC m=+32.069480711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbsfb" (UID: "7aad1c89-4805-4c51-a9dc-26db67c46f46") : secret "insights-runtime-extractor-tls" not found Oct 30 12:51:37.169491 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:37.169485 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-rbac-proxy-cm podName:7aad1c89-4805-4c51-a9dc-26db67c46f46 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:38.169477383 +0000 UTC m=+32.069506423 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hbsfb" (UID: "7aad1c89-4805-4c51-a9dc-26db67c46f46") : configmap "kube-rbac-proxy" not found Oct 30 12:51:37.169635 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.169500 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e7c1377-fb2f-4b5f-83a7-b799206fb344-metrics-tls\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:37.169635 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:37.169595 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:51:37.169635 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:37.169621 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e7c1377-fb2f-4b5f-83a7-b799206fb344-metrics-tls podName:6e7c1377-fb2f-4b5f-83a7-b799206fb344 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:38.169614643 +0000 UTC m=+32.069643684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6e7c1377-fb2f-4b5f-83a7-b799206fb344-metrics-tls") pod "dns-default-b7kx7" (UID: "6e7c1377-fb2f-4b5f-83a7-b799206fb344") : secret "dns-default-metrics-tls" not found Oct 30 12:51:37.181675 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.181641 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p8vd8"] Oct 30 12:51:37.181810 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.181693 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" Oct 30 12:51:37.184525 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:37.184503 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2e503d_c234_415b_bce5_ec542e09d50b.slice/crio-77484767ccf78143bd113a291395a2a537c06db202d0683795263696bdf856a4 WatchSource:0}: Error finding container 77484767ccf78143bd113a291395a2a537c06db202d0683795263696bdf856a4: Status 404 returned error can't find the container with id 77484767ccf78143bd113a291395a2a537c06db202d0683795263696bdf856a4 Oct 30 12:51:37.509955 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.509862 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-f4rcm"] Oct 30 12:51:37.512955 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:37.512909 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f771a8_7a67_429d_8701_9c3ffa29550c.slice/crio-ea007a860eb40b28fe0197ca25fce4239e44bc1c76e8d212b8063fe1c90db96f WatchSource:0}: Error finding container ea007a860eb40b28fe0197ca25fce4239e44bc1c76e8d212b8063fe1c90db96f: Status 404 returned error can't find the container with id ea007a860eb40b28fe0197ca25fce4239e44bc1c76e8d212b8063fe1c90db96f Oct 30 12:51:37.562836 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.562807 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:37.563008 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.562898 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:37.563008 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.562908 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:37.581056 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.581034 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 12:51:37.581397 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.581139 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 12:51:37.582750 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.582728 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 12:51:37.582750 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.582743 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-68fxr\"" Oct 30 12:51:37.582906 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.582762 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hszmd\"" Oct 30 12:51:37.582906 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.582786 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wqgtx\"" Oct 30 12:51:37.583025 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.582959 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 12:51:37.836020 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.835856 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" event={"ID":"d4f771a8-7a67-429d-8701-9c3ffa29550c","Type":"ContainerStarted","Data":"ea007a860eb40b28fe0197ca25fce4239e44bc1c76e8d212b8063fe1c90db96f"} Oct 30 12:51:37.837251 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:37.837194 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p8vd8" event={"ID":"1f2e503d-c234-415b-bce5-ec542e09d50b","Type":"ContainerStarted","Data":"77484767ccf78143bd113a291395a2a537c06db202d0683795263696bdf856a4"} Oct 30 12:51:38.178686 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:38.178611 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:38.178686 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:38.178675 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e7c1377-fb2f-4b5f-83a7-b799206fb344-metrics-tls\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:38.178851 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:38.178741 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:38.178851 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:38.178745 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 12:51:38.178851 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:38.178780 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:51:38.178851 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:38.178824 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-rbac-proxy-cm podName:7aad1c89-4805-4c51-a9dc-26db67c46f46 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:40.178796268 +0000 UTC m=+34.078825385 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hbsfb" (UID: "7aad1c89-4805-4c51-a9dc-26db67c46f46") : configmap "kube-rbac-proxy" not found Oct 30 12:51:38.178851 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:38.178841 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e7c1377-fb2f-4b5f-83a7-b799206fb344-metrics-tls podName:6e7c1377-fb2f-4b5f-83a7-b799206fb344 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:40.178834937 +0000 UTC m=+34.078863978 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6e7c1377-fb2f-4b5f-83a7-b799206fb344-metrics-tls") pod "dns-default-b7kx7" (UID: "6e7c1377-fb2f-4b5f-83a7-b799206fb344") : secret "dns-default-metrics-tls" not found Oct 30 12:51:38.178851 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:38.178848 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:51:38.179137 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:38.178881 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls podName:7aad1c89-4805-4c51-a9dc-26db67c46f46 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:40.178871051 +0000 UTC m=+34.078900117 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbsfb" (UID: "7aad1c89-4805-4c51-a9dc-26db67c46f46") : secret "insights-runtime-extractor-tls" not found Oct 30 12:51:39.389056 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:39.389022 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z6m2l_c10fc6a1-af69-48c2-b3dc-131e48f36b22/dns-node-resolver/0.log" Oct 30 12:51:40.188429 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:40.188362 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kvsqc_10b4d740-fb6d-486e-b6c1-5c6115a10496/node-ca/0.log" Oct 30 12:51:40.195638 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:40.195613 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e7c1377-fb2f-4b5f-83a7-b799206fb344-metrics-tls\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:40.195741 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:40.195699 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:40.195741 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:40.195730 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:40.195803 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:40.195767 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 12:51:40.195841 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:40.195812 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 12:51:40.195841 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:40.195822 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e7c1377-fb2f-4b5f-83a7-b799206fb344-metrics-tls podName:6e7c1377-fb2f-4b5f-83a7-b799206fb344 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:44.195805626 +0000 UTC m=+38.095834666 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6e7c1377-fb2f-4b5f-83a7-b799206fb344-metrics-tls") pod "dns-default-b7kx7" (UID: "6e7c1377-fb2f-4b5f-83a7-b799206fb344") : secret "dns-default-metrics-tls" not found Oct 30 12:51:40.195841 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:40.195824 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:51:40.195962 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:40.195845 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-rbac-proxy-cm podName:7aad1c89-4805-4c51-a9dc-26db67c46f46 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:44.195836042 +0000 UTC m=+38.095865096 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hbsfb" (UID: "7aad1c89-4805-4c51-a9dc-26db67c46f46") : configmap "kube-rbac-proxy" not found Oct 30 12:51:40.195962 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:40.195876 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls podName:7aad1c89-4805-4c51-a9dc-26db67c46f46 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:44.19586192 +0000 UTC m=+38.095890966 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbsfb" (UID: "7aad1c89-4805-4c51-a9dc-26db67c46f46") : secret "insights-runtime-extractor-tls" not found Oct 30 12:51:40.296317 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:40.296291 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs\") pod \"network-metrics-daemon-7l5g4\" (UID: \"6b553c14-1755-44ed-8dec-a2d1d8259c1d\") " pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:40.296460 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:40.296329 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:40.296514 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:40.296463 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 12:51:40.296549 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:40.296537 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs podName:6b553c14-1755-44ed-8dec-a2d1d8259c1d nodeName:}" failed. No retries permitted until 2026-10-30 12:52:12.296521725 +0000 UTC m=+66.196550767 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs") pod "network-metrics-daemon-7l5g4" (UID: "6b553c14-1755-44ed-8dec-a2d1d8259c1d") : secret "metrics-daemon-secret" not found Oct 30 12:51:40.298393 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:40.298364 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:40.353211 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:40.353178 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:40.397512 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:40.397478 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfnw\" (UniqueName: \"kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw\") pod \"network-check-target-xb4w4\" (UID: \"30b22e49-8949-4dc1-9886-cc25b97f3436\") " pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:40.400091 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:40.400064 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lfnw\" (UniqueName: \"kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw\") pod \"network-check-target-xb4w4\" (UID: \"30b22e49-8949-4dc1-9886-cc25b97f3436\") " pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:40.601352 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:40.601264 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:40.671376 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:40.671347 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r5lrn"] Oct 30 12:51:40.675656 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:40.675628 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f27309_51ca_4116_b0e9_10cf9e144c36.slice/crio-399d3d74faa052e4b30191ceeddfd37a932f6041b1a83399643ca8792d89b308 WatchSource:0}: Error finding container 399d3d74faa052e4b30191ceeddfd37a932f6041b1a83399643ca8792d89b308: Status 404 returned error can't find the container with id 399d3d74faa052e4b30191ceeddfd37a932f6041b1a83399643ca8792d89b308 Oct 30 12:51:40.846457 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:40.846418 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" event={"ID":"d4f771a8-7a67-429d-8701-9c3ffa29550c","Type":"ContainerStarted","Data":"b5bb8410ec62d4507df2369efd622d3b88d82d956852e79dd2e9084f25f4754b"} Oct 30 12:51:40.848084 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:40.848054 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p8vd8" event={"ID":"1f2e503d-c234-415b-bce5-ec542e09d50b","Type":"ContainerStarted","Data":"3816d7bb642354dda826ec2c73da95d680c536d59eb2e925531af24e0c610ef6"} Oct 30 12:51:40.849200 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:40.849177 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r5lrn" event={"ID":"87f27309-51ca-4116-b0e9-10cf9e144c36","Type":"ContainerStarted","Data":"399d3d74faa052e4b30191ceeddfd37a932f6041b1a83399643ca8792d89b308"} Oct 30 12:51:40.890132 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:40.890047 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" podStartSLOduration=2.174096248 podStartE2EDuration="4.890031267s" totalImagesPullingTime="2.715935019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:40.889233589 +0000 UTC m=+34.789262651" watchObservedRunningTime="2026-10-30 12:51:40.890031267 +0000 UTC m=+34.790060375" Oct 30 12:51:40.928027 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:40.927999 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xb4w4"] Oct 30 12:51:40.931174 ip-10-0-130-200 kubenswrapper[2581]: W1030 12:51:40.931147 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b22e49_8949_4dc1_9886_cc25b97f3436.slice/crio-b0207c497a7ba8391419a0401bee6de8bd19a2237c76204a21a018525023c7e7 WatchSource:0}: Error finding container b0207c497a7ba8391419a0401bee6de8bd19a2237c76204a21a018525023c7e7: Status 404 returned error can't find the container with id b0207c497a7ba8391419a0401bee6de8bd19a2237c76204a21a018525023c7e7 Oct 30 12:51:40.932147 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:40.932102 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-p8vd8" podStartSLOduration=1.878284252 podStartE2EDuration="4.932085755s" totalImagesPullingTime="3.053801503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:40.931310369 +0000 UTC m=+34.831339432" watchObservedRunningTime="2026-10-30 12:51:40.932085755 +0000 UTC m=+34.832114819" Oct 30 12:51:40.988795 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:40.988770 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p8vd8_1f2e503d-c234-415b-bce5-ec542e09d50b/serve-healthcheck-canary/0.log" Oct 30 12:51:41.854430 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:41.854386 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xb4w4" event={"ID":"30b22e49-8949-4dc1-9886-cc25b97f3436","Type":"ContainerStarted","Data":"b0207c497a7ba8391419a0401bee6de8bd19a2237c76204a21a018525023c7e7"} Oct 30 12:51:44.232539 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:44.232493 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e7c1377-fb2f-4b5f-83a7-b799206fb344-metrics-tls\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:44.233060 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:44.232563 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:44.233060 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:44.232585 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:44.233060 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:44.232749 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:51:44.233060 ip-10-0-130-200 kubenswrapper[2581]: E1030 12:51:44.232818 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls podName:7aad1c89-4805-4c51-a9dc-26db67c46f46 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:52.232803186 +0000 UTC m=+46.132832239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbsfb" (UID: "7aad1c89-4805-4c51-a9dc-26db67c46f46") : secret "insights-runtime-extractor-tls" not found Oct 30 12:51:44.233264 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:44.233193 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:44.235065 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:44.235041 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e7c1377-fb2f-4b5f-83a7-b799206fb344-metrics-tls\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:44.302909 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:44.302873 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:44.742048 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:44.742016 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b7kx7"] Oct 30 12:51:44.875229 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:44.875193 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b7kx7" event={"ID":"6e7c1377-fb2f-4b5f-83a7-b799206fb344","Type":"ContainerStarted","Data":"8a493222c8d2e4ae9906d35f1657665a7128208f954db15c7718e9cb1968ae67"} Oct 30 12:51:44.876871 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:44.876839 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xb4w4" event={"ID":"30b22e49-8949-4dc1-9886-cc25b97f3436","Type":"ContainerStarted","Data":"2633f48e71308404c7edc362712277d9c4caf4c97916a073b25ca4ad27b2f514"} Oct 30 12:51:44.877024 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:44.876976 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:44.930845 ip-10-0-130-200 kubenswrapper[2581]: I1030 12:51:44.930783 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xb4w4" podStartSLOduration=34.424172014 podStartE2EDuration="37.93076342s" totalImagesPullingTime="3.506591406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:44.930045602 +0000 UTC m=+38.830074666" watchObservedRunningTime="2026-10-30 12:51:44.93076342 +0000 UTC m=+38.830792481" Oct 30 12:51:46.111638 ip-10-0-130-200 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 12:51:46.127655 ip-10-0-130-200 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 12:51:46.127845 ip-10-0-130-200 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 12:51:46.127900 ip-10-0-130-200 systemd[1]: kubelet.service: Consumed 2.370s CPU time. Oct 30 12:51:46.138598 ip-10-0-130-200 systemd[1]: Starting Kubernetes Kubelet... Oct 30 12:51:46.255321 ip-10-0-130-200 kubenswrapper[4503]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:51:46.255321 ip-10-0-130-200 kubenswrapper[4503]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 12:51:46.255321 ip-10-0-130-200 kubenswrapper[4503]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:51:46.255321 ip-10-0-130-200 kubenswrapper[4503]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:51:46.258898 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.258791 4503 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:46.258898 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.258896 4503 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:46.258898 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.258904 4503 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:46.258898 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.258909 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:46.259150 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.258914 4503 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:46.259150 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.258919 4503 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:46.259150 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.258924 4503 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:46.259150 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.258928 4503 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:46.259150 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.258933 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:46.259150 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.258952 4503 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:46.259150 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.258958 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:46.259150 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.258963 4503 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:46.259150 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.258967 4503 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:46.259150 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.258973 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:46.259150 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.258979 4503 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:46.259150 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.258986 4503 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:46.259150 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.258991 4503 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:46.259150 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.258996 4503 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:46.259150 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259001 4503 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:46.259150 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259004 4503 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:46.259150 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259007 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:46.259150 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259009 4503 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:46.259150 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259019 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259022 4503 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259025 4503 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259028 4503 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259032 4503 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259034 4503 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259039 4503 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259045 4503 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259049 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259054 4503 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259059 4503 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259063 4503 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259068 4503 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259073 4503 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259076 4503 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259079 4503 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259083 4503 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259088 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259093 4503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259097 4503 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:46.259907 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259103 4503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259108 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259112 4503 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259116 4503 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259122 4503 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259127 4503 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259130 4503 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259133 4503 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259136 4503 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259139 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259142 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259145 4503 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259148 4503 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259150 4503 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259153 4503 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259160 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259167 4503 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259171 4503 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259176 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259180 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:46.260705 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259184 4503 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:46.261422 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259188 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:46.261422 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259192 4503 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:46.261422 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259196 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:46.261422 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259200 4503 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:46.261422 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259205 4503 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:46.261422 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259209 4503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:46.261422 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259213 4503 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:46.261422 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259218 4503 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:46.261422 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259223 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:46.261422 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259228 4503 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:46.261422 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259234 4503 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:46.261422 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259238 4503 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:46.261422 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259243 4503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:46.261422 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259247 4503 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:46.261422 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259251 4503 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:46.261422 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259255 4503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:46.261422 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259260 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:46.261422 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259264 4503 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:46.261422 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259268 4503 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259273 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259277 4503 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259283 4503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259288 4503 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259294 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259299 4503 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259305 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259311 4503 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259314 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259319 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259323 4503 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259328 4503 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259333 4503 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259341 4503 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259348 4503 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259353 4503 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259356 4503 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259360 4503 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259366 4503 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:46.262119 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259370 4503 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259376 4503 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259383 4503 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259387 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.259392 4503 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260561 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260572 4503 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260575 4503 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260579 4503 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260582 4503 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260586 4503 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260588 4503 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260591 4503 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260595 4503 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260601 4503 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260605 4503 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260610 4503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260614 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260619 4503 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260623 4503 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:46.262799 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260626 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260629 4503 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260632 4503 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260635 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260638 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260641 4503 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260643 4503 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260646 4503 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260649 4503 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260652 4503 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260658 4503 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260662 4503 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260667 4503 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260672 4503 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260677 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260681 4503 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260686 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260698 4503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260703 4503 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260708 4503 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:46.263502 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260713 4503 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260717 4503 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260721 4503 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260725 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260730 4503 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260734 4503 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260738 4503 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260742 4503 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260746 4503 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260750 4503 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260755 4503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260764 4503 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260769 4503 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260773 4503 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260778 4503 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260782 4503 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260784 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260787 4503 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260790 4503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260793 4503 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:46.264155 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260795 4503 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:46.264793 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260798 4503 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:46.264793 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260801 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:46.264793 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260803 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:46.264793 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260805 4503 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:46.264793 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260808 4503 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:46.264793 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260812 4503 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:46.264793 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260816 4503 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:46.264793 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260820 4503 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:46.264793 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260824 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:46.264793 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260834 4503 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:46.264793 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260838 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:46.264793 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260843 4503 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:46.264793 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260848 4503 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:46.264793 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260852 4503 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:46.264793 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260855 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:46.264793 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260858 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:46.264793 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260863 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:46.264793 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260865 4503 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:46.264793 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260868 4503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:46.265401 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260871 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:46.265401 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260874 4503 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:46.265401 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260876 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:46.265401 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260879 4503 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:46.265401 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260882 4503 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:46.265401 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260885 4503 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:46.265401 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260888 4503 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:46.265401 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260890 4503 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:46.265401 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260893 4503 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:46.265401 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260896 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:46.265401 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260902 4503 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:46.265401 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260908 4503 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:46.265401 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260911 4503 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:46.265401 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260915 4503 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:46.265401 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260917 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:46.265401 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260920 4503 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:46.265401 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260923 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:46.265401 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260926 4503 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:46.265401 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260929 4503 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:46.266043 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260931 4503 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:46.266043 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260934 4503 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:46.266043 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260952 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:46.266043 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260955 4503 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:46.266043 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260960 4503 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:46.266043 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260964 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:46.266043 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260967 4503 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:46.266043 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260971 4503 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:46.266043 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260973 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:46.266043 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260976 4503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:46.266043 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260978 4503 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:46.266043 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.260981 4503 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:46.266043 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261620 4503 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:46.266043 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261631 4503 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:46.266043 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261634 4503 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:46.266043 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261637 4503 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:46.266043 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261639 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:46.266043 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261642 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:46.266043 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261645 4503 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261648 4503 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261651 4503 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261654 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261657 4503 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261659 4503 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261662 4503 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261665 4503 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261667 4503 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261670 4503 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261672 4503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261675 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261678 4503 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261682 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261686 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261690 4503 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261695 4503 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261699 4503 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261705 4503 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261709 4503 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:46.266646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261714 4503 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261718 4503 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261724 4503 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261728 4503 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261733 4503 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261737 4503 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261741 4503 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261747 4503 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261751 4503 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261756 4503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261760 4503 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261764 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261766 4503 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261769 4503 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261772 4503 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261775 4503 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261777 4503 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261780 4503 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261783 4503 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261786 4503 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:46.267239 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261788 4503 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:46.267860 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261791 4503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:46.267860 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261794 4503 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:46.267860 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261796 4503 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:46.267860 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261799 4503 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:46.267860 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261802 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:46.267860 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261805 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:46.267860 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261807 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:46.267860 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261810 4503 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:46.267860 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261813 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:46.267860 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261815 4503 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:46.267860 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261818 4503 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:46.267860 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261821 4503 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:46.267860 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261824 4503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:46.267860 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261829 4503 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:46.267860 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261835 4503 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:46.267860 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261840 4503 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:46.267860 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261845 4503 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:46.267860 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261861 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:46.267860 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261868 4503 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261873 4503 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261876 4503 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261879 4503 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261882 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261884 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261887 4503 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261890 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261892 4503 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261895 4503 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261898 4503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261900 4503 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261903 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261906 4503 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261909 4503 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261912 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261914 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261917 4503 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261921 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261924 4503 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:46.268510 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261926 4503 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:46.269204 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261929 4503 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:46.269204 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261932 4503 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:46.269204 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261948 4503 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:46.269204 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261952 4503 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:46.269204 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261964 4503 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:46.269204 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261967 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:46.269204 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261972 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:46.269204 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261976 4503 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:46.269204 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261980 4503 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:46.269204 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261985 4503 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:46.269204 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261991 4503 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:46.269204 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.261999 4503 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:46.269204 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.262005 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:46.269204 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.262010 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:46.269204 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.262026 4503 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:46.269204 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.262031 4503 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:46.269204 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.262036 4503 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:46.269204 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.262040 4503 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:46.269204 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.262044 4503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.262048 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262197 4503 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262209 4503 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262219 4503 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262227 4503 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262235 4503 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262240 4503 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262245 4503 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262250 4503 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262255 4503 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262264 4503 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262270 4503 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262275 4503 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262280 4503 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262286 4503 flags.go:64] FLAG: --cgroup-root="" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262292 4503 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262297 4503 flags.go:64] FLAG: --client-ca-file="" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262301 4503 flags.go:64] FLAG: --cloud-provider="external" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262315 4503 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262329 4503 flags.go:64] FLAG: --cluster-domain="" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262334 4503 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262338 4503 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262342 4503 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 12:51:46.269794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262345 4503 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262351 4503 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262354 4503 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262357 4503 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262361 4503 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262366 4503 flags.go:64] FLAG: --contention-profiling="false" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262371 4503 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262375 4503 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262380 4503 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262385 4503 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262392 4503 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262397 4503 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262402 4503 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262407 4503 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262413 4503 flags.go:64] FLAG: --enable-server="true" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262417 4503 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262429 4503 flags.go:64] FLAG: --event-burst="100" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262435 4503 flags.go:64] FLAG: --event-qps="50" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262440 4503 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262447 4503 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262450 4503 flags.go:64] FLAG: --eviction-hard="" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262455 4503 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262458 4503 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262461 4503 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262464 4503 flags.go:64] FLAG: --eviction-soft="" Oct 30 12:51:46.270639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262468 4503 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262471 4503 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262474 4503 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262478 4503 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262481 4503 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262483 4503 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262486 4503 flags.go:64] FLAG: --feature-gates="" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262491 4503 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262494 4503 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262500 4503 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262505 4503 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262510 4503 flags.go:64] FLAG: --healthz-port="10248" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262515 4503 flags.go:64] FLAG: --help="false" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262520 4503 flags.go:64] FLAG: --hostname-override="ip-10-0-130-200.ec2.internal" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262525 4503 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262530 4503 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262536 4503 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262541 4503 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262547 4503 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262552 4503 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262556 4503 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262560 4503 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262563 4503 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 12:51:46.271689 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262566 4503 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262569 4503 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262572 4503 flags.go:64] FLAG: --kube-reserved="" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262578 4503 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262581 4503 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262585 4503 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262588 4503 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262590 4503 flags.go:64] FLAG: --lock-file="" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262593 4503 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262597 4503 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262600 4503 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262607 4503 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262612 4503 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262618 4503 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262622 4503 flags.go:64] FLAG: --logging-format="text" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262627 4503 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262632 4503 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262636 4503 flags.go:64] FLAG: --manifest-url="" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262641 4503 flags.go:64] FLAG: --manifest-url-header="" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262647 4503 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262652 4503 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262658 4503 flags.go:64] FLAG: --max-pods="110" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262661 4503 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262664 4503 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262668 4503 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 12:51:46.272752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262671 4503 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262674 4503 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262677 4503 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262679 4503 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262691 4503 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262696 4503 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262701 4503 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262704 4503 flags.go:64] FLAG: --pod-cidr="" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262707 4503 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262710 4503 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262715 4503 flags.go:64] FLAG: --pods-per-core="0" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262718 4503 flags.go:64] FLAG: --port="10250" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262721 4503 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262724 4503 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cdf3b5499be48d54" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262727 4503 flags.go:64] FLAG: --qos-reserved="" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262730 4503 flags.go:64] FLAG: --read-only-port="10255" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262733 4503 flags.go:64] FLAG: --register-node="true" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262736 4503 flags.go:64] FLAG: --register-with-taints="" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262743 4503 flags.go:64] FLAG: --registry-burst="10" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262746 4503 flags.go:64] FLAG: --registry-qps="5" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262749 4503 flags.go:64] FLAG: --reserved-cpus="" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262753 4503 flags.go:64] FLAG: --reserved-memory="" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262757 4503 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262761 4503 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262766 4503 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262771 4503 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 12:51:46.273576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262776 4503 flags.go:64] FLAG: --runonce="false" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262780 4503 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262785 4503 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262790 4503 flags.go:64] FLAG: --seccomp-default="false" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262794 4503 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262799 4503 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262803 4503 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262808 4503 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262813 4503 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262818 4503 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262821 4503 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262826 4503 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262831 4503 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262840 4503 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262845 4503 flags.go:64] FLAG: --system-cgroups="" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262850 4503 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262862 4503 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262867 4503 flags.go:64] FLAG: --tls-cert-file="" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262872 4503 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262880 4503 flags.go:64] FLAG: --tls-min-version="" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262883 4503 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262886 4503 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262889 4503 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262892 4503 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262895 4503 flags.go:64] FLAG: --v="2" Oct 30 12:51:46.274351 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262902 4503 flags.go:64] FLAG: --version="false" Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262909 4503 flags.go:64] FLAG: --vmodule="" Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262915 4503 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.262921 4503 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263126 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263132 4503 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263136 4503 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263140 4503 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263143 4503 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263146 4503 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263149 4503 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263152 4503 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263156 4503 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263158 4503 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263161 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263164 4503 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263167 4503 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263169 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263172 4503 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263175 4503 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:46.275321 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263179 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:46.276054 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263182 4503 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:46.276054 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263187 4503 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:46.276054 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263194 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:46.276054 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263201 4503 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:46.276054 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263208 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:46.276054 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263213 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:46.276054 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263217 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:46.276054 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263222 4503 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:46.276054 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263226 4503 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:46.276054 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263230 4503 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:46.276054 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263235 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:46.276054 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263239 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:46.276054 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263243 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:46.276054 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263248 4503 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:46.276054 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263254 4503 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:46.276054 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263259 4503 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:46.276054 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263263 4503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:46.276054 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263268 4503 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:46.276054 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263273 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:46.276752 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263278 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:46.276752 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263283 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:46.276752 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263287 4503 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:46.276752 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263291 4503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:46.276752 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263293 4503 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:46.276752 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263296 4503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:46.276752 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263298 4503 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:46.276752 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263301 4503 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:46.276752 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263304 4503 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:46.276752 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263306 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:46.276752 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263309 4503 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:46.276752 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263312 4503 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:46.276752 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263316 4503 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:46.276752 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263319 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:46.276752 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263322 4503 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:46.276752 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263327 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:46.276752 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263332 4503 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:46.276752 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263336 4503 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:46.276752 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263340 4503 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263344 4503 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263348 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263351 4503 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263356 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263360 4503 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263365 4503 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263369 4503 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263372 4503 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263374 4503 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263377 4503 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263382 4503 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263386 4503 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263391 4503 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263395 4503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263399 4503 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263402 4503 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263404 4503 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263407 4503 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263410 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:46.277749 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263412 4503 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263415 4503 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263418 4503 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263420 4503 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263424 4503 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263426 4503 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263431 4503 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263433 4503 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263436 4503 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263440 4503 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263443 4503 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263447 4503 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263451 4503 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263456 4503 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263460 4503 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263464 4503 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263468 4503 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263472 4503 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263476 4503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263480 4503 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:46.278402 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263484 4503 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:46.279073 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263488 4503 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:46.279073 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263493 4503 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:46.279073 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263497 4503 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:46.279073 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263501 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:46.279073 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263506 4503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:46.279073 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263521 4503 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:46.279073 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263526 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:46.279073 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263530 4503 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:46.279073 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263534 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:46.279073 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263538 4503 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:46.279073 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263543 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:46.279073 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.263576 4503 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:51:46.279073 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263629 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:46.279073 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263636 4503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:46.279073 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263641 4503 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263645 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263650 4503 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263654 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263658 4503 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263661 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263665 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263668 4503 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263671 4503 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263673 4503 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263676 4503 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263678 4503 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263681 4503 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263684 4503 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263686 4503 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263689 4503 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263692 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263695 4503 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263697 4503 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263700 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:46.279634 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263702 4503 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:46.280339 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263705 4503 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:46.280339 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263709 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:46.280339 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263712 4503 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:46.280339 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263716 4503 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:46.280339 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263720 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:46.280339 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263726 4503 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:46.280339 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263734 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:46.280339 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263738 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:46.280339 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263744 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:46.280339 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263748 4503 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:46.280339 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263752 4503 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:46.280339 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263756 4503 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:46.280339 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263761 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:46.280339 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263765 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:46.280339 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263768 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:46.280339 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263773 4503 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:46.280339 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263776 4503 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:46.280339 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263778 4503 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:46.280339 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263783 4503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:46.281109 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263787 4503 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:46.281109 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263792 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:46.281109 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263797 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:46.281109 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263801 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:46.281109 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263805 4503 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:46.281109 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263810 4503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:46.281109 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263815 4503 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:46.281109 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263820 4503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:46.281109 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263825 4503 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:46.281109 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263829 4503 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:46.281109 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263834 4503 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:46.281109 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263838 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:46.281109 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263842 4503 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:46.281109 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263847 4503 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:46.281109 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263851 4503 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:46.281109 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263855 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:46.281109 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263859 4503 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:46.281109 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263863 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:46.281109 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263867 4503 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263871 4503 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263875 4503 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263880 4503 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263883 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263885 4503 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263890 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263893 4503 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263895 4503 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263898 4503 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263905 4503 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263910 4503 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263914 4503 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263918 4503 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263923 4503 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263929 4503 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263947 4503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263952 4503 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263957 4503 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263961 4503 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:46.281783 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263966 4503 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263970 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263974 4503 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263978 4503 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263983 4503 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263987 4503 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263991 4503 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263995 4503 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.263999 4503 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.264002 4503 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.264005 4503 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.264008 4503 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.264011 4503 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.264014 4503 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.264016 4503 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.264021 4503 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.264026 4503 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.264031 4503 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.264035 4503 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.264040 4503 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:46.282452 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.264044 4503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:46.283159 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.264049 4503 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:46.283159 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.264056 4503 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:46.283159 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.264060 4503 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:46.283159 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.264063 4503 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:46.283159 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.264066 4503 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:46.283159 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.268432 4503 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 12:51:46.283159 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.268447 4503 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 12:51:46.283159 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.268466 4503 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 12:51:46.283159 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.268472 4503 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 12:51:46.283159 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268539 4503 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:46.283159 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268547 4503 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:46.283159 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268552 4503 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:46.283159 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268557 4503 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:46.283159 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268562 4503 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:46.283159 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268569 4503 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:46.283159 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268576 4503 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:46.283159 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268586 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:46.283159 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268592 4503 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:46.283159 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268597 4503 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268602 4503 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268607 4503 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268613 4503 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268617 4503 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268622 4503 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268627 4503 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268632 4503 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268635 4503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268638 4503 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268641 4503 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268645 4503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268650 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268654 4503 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268659 4503 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268663 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268668 4503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268673 4503 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268677 4503 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268681 4503 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:46.283855 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268685 4503 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:46.284566 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268691 4503 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:46.284566 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268694 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:46.284566 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268697 4503 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:46.284566 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268700 4503 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:46.284566 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268702 4503 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:46.284566 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268705 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:46.284566 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268708 4503 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:46.284566 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268710 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:46.284566 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268713 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:46.284566 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268716 4503 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:46.284566 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268718 4503 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:46.284566 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268721 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:46.284566 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268724 4503 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:46.284566 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268726 4503 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:46.284566 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268729 4503 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:46.284566 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268734 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:46.284566 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268737 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:46.284566 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268741 4503 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:46.284566 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268746 4503 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:46.285369 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268750 4503 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:46.285369 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268753 4503 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:46.285369 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268761 4503 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:46.285369 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268766 4503 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:46.285369 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268771 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:46.285369 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268785 4503 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:46.285369 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268790 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:46.285369 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268794 4503 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:46.285369 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268797 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:46.285369 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268800 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:46.285369 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268803 4503 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:46.285369 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268805 4503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:46.285369 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268808 4503 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:46.285369 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268811 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:46.285369 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268814 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:46.285369 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268817 4503 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:46.285369 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268819 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:46.285369 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268822 4503 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:46.285369 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268824 4503 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268827 4503 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268830 4503 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268833 4503 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268837 4503 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268842 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268846 4503 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268850 4503 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268854 4503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268859 4503 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268863 4503 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268868 4503 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268872 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268876 4503 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268880 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268884 4503 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268888 4503 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268893 4503 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268897 4503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268902 4503 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:46.285973 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268905 4503 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:46.286710 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268908 4503 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:46.286710 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268910 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:46.286710 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268913 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:46.286710 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268916 4503 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:46.286710 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268918 4503 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:46.286710 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268921 4503 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:46.286710 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268924 4503 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:46.286710 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268927 4503 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:46.286710 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268930 4503 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:46.286710 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268932 4503 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:46.286710 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268949 4503 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:46.286710 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268954 4503 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:46.286710 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268958 4503 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:46.286710 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268963 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:46.286710 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268968 4503 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:46.286710 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268972 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:46.286710 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268976 4503 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:46.286710 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.268980 4503 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:46.287428 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.269020 4503 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:51:46.287428 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269180 4503 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:46.287428 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269185 4503 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:46.287428 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269188 4503 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:46.287428 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269191 4503 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:46.287428 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269194 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:46.287428 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269199 4503 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:46.287428 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269203 4503 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:46.287428 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269206 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:46.287428 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269209 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:46.287428 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269212 4503 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:46.287428 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269215 4503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:46.287428 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269218 4503 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:46.287428 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269221 4503 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:46.287428 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269226 4503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269230 4503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269235 4503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269239 4503 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269243 4503 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269247 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269251 4503 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269255 4503 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269260 4503 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269264 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269269 4503 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269273 4503 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269277 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269281 4503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269284 4503 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269287 4503 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269289 4503 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269292 4503 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269295 4503 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269298 4503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:46.287998 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269300 4503 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269303 4503 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269306 4503 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269308 4503 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269312 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269315 4503 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269317 4503 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269320 4503 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269322 4503 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269325 4503 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269331 4503 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269335 4503 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269340 4503 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269344 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269348 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269352 4503 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269356 4503 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269360 4503 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269364 4503 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269368 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:46.288691 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269373 4503 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269377 4503 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269382 4503 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269385 4503 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269389 4503 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269391 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269394 4503 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269397 4503 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269399 4503 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269402 4503 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269405 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269407 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269410 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269412 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269415 4503 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269417 4503 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269420 4503 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269423 4503 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269426 4503 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269429 4503 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:46.289411 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269431 4503 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269434 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269438 4503 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269440 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269443 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269446 4503 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269448 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269451 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269454 4503 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269456 4503 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269459 4503 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269461 4503 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269464 4503 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269467 4503 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269471 4503 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269475 4503 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269479 4503 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269483 4503 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269487 4503 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269491 4503 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:46.290091 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269495 4503 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:46.290775 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269514 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:46.290775 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269517 4503 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:46.290775 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269520 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:46.290775 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269524 4503 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:46.290775 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269529 4503 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:46.290775 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269534 4503 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:46.290775 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269539 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:46.290775 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269543 4503 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:46.290775 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269547 4503 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:46.290775 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269553 4503 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:46.290775 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269558 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:46.290775 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269564 4503 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:46.290775 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.269598 4503 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:51:46.290775 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269678 4503 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:46.290775 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269687 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:46.291414 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269692 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:46.291414 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269695 4503 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:46.291414 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269698 4503 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:46.291414 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269701 4503 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:46.291414 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269703 4503 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:46.291414 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269706 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:46.291414 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269709 4503 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:46.291414 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269712 4503 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:46.291414 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269716 4503 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:46.291414 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269721 4503 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:46.291414 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269725 4503 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:46.291414 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269729 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:46.291414 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269732 4503 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:46.291414 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269737 4503 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:46.291414 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269741 4503 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:46.291414 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269746 4503 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:46.291414 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269750 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:46.291414 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269754 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:46.291414 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269758 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269762 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269766 4503 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269770 4503 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269775 4503 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269779 4503 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269784 4503 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269788 4503 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269793 4503 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269799 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269805 4503 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269810 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269816 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269819 4503 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269823 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269825 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269828 4503 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269831 4503 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269834 4503 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269836 4503 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:46.292089 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269839 4503 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:46.292904 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269842 4503 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:46.292904 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269844 4503 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:46.292904 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269848 4503 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:46.292904 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269851 4503 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:46.292904 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269854 4503 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:46.292904 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269858 4503 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:46.292904 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269863 4503 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:46.292904 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269867 4503 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:46.292904 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269871 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:46.292904 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269875 4503 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:46.292904 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269879 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:46.292904 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269883 4503 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:46.292904 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269887 4503 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:46.292904 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269891 4503 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:46.292904 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269896 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:46.292904 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269901 4503 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:46.292904 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269904 4503 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:46.292904 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269907 4503 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:46.292904 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269911 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:46.293820 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269915 4503 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:46.293820 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269920 4503 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:46.293820 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269925 4503 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:46.293820 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269930 4503 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:46.293820 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269950 4503 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:46.293820 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269955 4503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:46.293820 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269962 4503 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:46.293820 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269966 4503 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:46.293820 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269969 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:46.293820 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269972 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:46.293820 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269974 4503 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:46.293820 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269977 4503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:46.293820 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269980 4503 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:46.293820 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269982 4503 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:46.293820 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269985 4503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:46.293820 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269988 4503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:46.293820 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269992 4503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:46.293820 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.269996 4503 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:46.293820 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270000 4503 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270004 4503 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270008 4503 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270013 4503 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270017 4503 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270020 4503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270025 4503 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270029 4503 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270033 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270038 4503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270041 4503 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270044 4503 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270047 4503 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270049 4503 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270054 4503 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270058 4503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270063 4503 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270068 4503 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270075 4503 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270081 4503 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:46.294595 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270087 4503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:46.295454 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270091 4503 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:46.295454 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270095 4503 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:46.295454 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270098 4503 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:46.295454 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270101 4503 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:46.295454 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270103 4503 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:46.295454 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270106 4503 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:46.295454 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:46.270110 4503 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:46.295454 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.270185 4503 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 12:51:46.295454 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.272564 4503 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 12:51:46.295454 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.272638 4503 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 12:51:46.295454 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.272875 4503 server.go:1014] "Starting client certificate rotation" Oct 30 12:51:46.295454 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.273028 4503 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 12:51:46.295454 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.273100 4503 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:46:06 +0000 UTC" deadline="2028-07-31 08:43:50.073029206 +0000 UTC" Oct 30 12:51:46.295454 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.273132 4503 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15355h52m3.799899225s" Oct 30 12:51:46.295454 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.273559 4503 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:51:46.295454 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.275186 4503 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:51:46.296223 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.277034 4503 log.go:25] "Validated CRI v1 runtime API" Oct 30 12:51:46.296223 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.278824 4503 log.go:25] "Validated CRI v1 image API" Oct 30 12:51:46.296223 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.279741 4503 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 12:51:46.296223 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.281971 4503 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c2c122e1-0639-4b96-aa24-52adb3b746a3:/dev/nvme0n1p4 d46bed61-ebbd-4ad9-97e3-86f7c526e1e8:/dev/nvme0n1p3] Oct 30 12:51:46.296412 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.281996 4503 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/211fcceeb7cccbd0d573b4e75e74d6a9601b07a8b2d77d206e690a48baac2320/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/211fcceeb7cccbd0d573b4e75e74d6a9601b07a8b2d77d206e690a48baac2320/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a889a230f994cb2ef99626cc18f96c135e3a0b2ede65a2a71b0d07f3b0fa098/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a889a230f994cb2ef99626cc18f96c135e3a0b2ede65a2a71b0d07f3b0fa098/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/399d3d74faa052e4b30191ceeddfd37a932f6041b1a83399643ca8792d89b308/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/399d3d74faa052e4b30191ceeddfd37a932f6041b1a83399643ca8792d89b308/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4daea0f7cc943381817be60dcec8a8a9ae8e6ab8b33d6217107fc9094900e753/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4daea0f7cc943381817be60dcec8a8a9ae8e6ab8b33d6217107fc9094900e753/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52bf0edd4be66970dd5081a5a0f4864dec438a44a47223a8155d2a4541e8644f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52bf0edd4be66970dd5081a5a0f4864dec438a44a47223a8155d2a4541e8644f/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f9d5919fe698693d8dd529bcb2b5d881abb7d72aa714248cc325c884212522d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f9d5919fe698693d8dd529bcb2b5d881abb7d72aa714248cc325c884212522d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77484767ccf78143bd113a291395a2a537c06db202d0683795263696bdf856a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77484767ccf78143bd113a291395a2a537c06db202d0683795263696bdf856a4/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a493222c8d2e4ae9906d35f1657665a7128208f954db15c7718e9cb1968ae67/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a493222c8d2e4ae9906d35f1657665a7128208f954db15c7718e9cb1968ae67/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8acf9e7ab72f0281efb81256a559ac1c739935c6d5c72042bf94bfb50d8aecf5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8acf9e7ab72f0281efb81256a559ac1c739935c6d5c72042bf94bfb50d8aecf5/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0207c497a7ba8391419a0401bee6de8bd19a2237c76204a21a018525023c7e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0207c497a7ba8391419a0401bee6de8bd19a2237c76204a21a018525023c7e7/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbab68617082441e7f1bef4668aec1d37f6d0dd7bf7596f51602340971802b93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbab68617082441e7f1bef4668aec1d37f6d0dd7bf7596f51602340971802b93/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6011e018859e7d10b70f4ea2d1750e0b8e068413d0e24e89b6e52e7cce50f11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6011e018859e7d10b70f4ea2d1750e0b8e068413d0e24e89b6e52e7cce50f11/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de47d77f14791de8fa895481fe59f9832817859f428d48da0c3c39fcb5396a92/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de47d77f14791de8fa895481fe59f9832817859f428d48da0c3c39fcb5396a92/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea007a860eb40b28fe0197ca25fce4239e44bc1c76e8d212b8063fe1c90db96f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea007a860eb40b28fe0197ca25fce4239e44bc1c76e8d212b8063fe1c90db96f/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9217d2105f8c59ba473a15b0058a619e3a3de93310651204fc5c42c6dab6dc8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9217d2105f8c59ba473a15b0058a619e3a3de93310651204fc5c42c6dab6dc8/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/061dda5b-09a3-493d-9c16-6d25055d5b82/volumes/kubernetes.io~projected/kube-api-access-qpmwj:{mountpoint:/var/lib/kubelet/pods/061dda5b-09a3-493d-9c16-6d25055d5b82/volumes/kubernetes.io~projected/kube-api-access-qpmwj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10b4d740-fb6d-486e-b6c1-5c6115a10496/volumes/kubernetes.io~projected/kube-api-access-ndhbb:{mountpoint:/var/lib/kubelet/pods/10b4d740-fb6d-486e-b6c1-5c6115a10496/volumes/kubernetes.io~projected/kube-api-access-ndhbb major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f2e503d-c234-415b-bce5-ec542e09d50b/volumes/kubernetes.io~projected/kube-api-access-bldsp:{mountpoint:/var/lib/kubelet/pods/1f2e503d-c234-415b-bce5-ec542e09d50b/volumes/kubernetes.io~projected/kube-api-access-bldsp major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f2e503d-c234-415b-bce5-ec542e09d50b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/1f2e503d-c234-415b-bce5-ec542e09d50b/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30b22e49-8949-4dc1-9886-cc25b97f3436/volumes/kubernetes.io~projected/kube-api-access-2lfnw:{mountpoint:/var/lib/kubelet/pods/30b22e49-8949-4dc1-9886-cc25b97f3436/volumes/kubernetes.io~projected/kube-api-access-2lfnw major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51f1ecdc-2207-4ac7-9193-77f917cc0143/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/51f1ecdc-2207-4ac7-9193-77f917cc0143/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b553c14-1755-44ed-8dec-a2d1d8259c1d/volumes/kubernetes.io~projected/kube-api-access-f7gmw:{mountpoint:/var/lib/kubelet/pods/6b553c14-1755-44ed-8dec-a2d1d8259c1d/volumes/kubernetes.io~projected/kube-api-access-f7gmw major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e7c1377-fb2f-4b5f-83a7-b799206fb344/volumes/kubernetes.io~projected/kube-api-access-zskvb:{mountpoint:/var/lib/kubelet/pods/6e7c1377-fb2f-4b5f-83a7-b799206fb344/volumes/kubernetes.io~projected/kube-api-access-zskvb major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e7c1377-fb2f-4b5f-83a7-b799206fb344/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/6e7c1377-fb2f-4b5f-83a7-b799206fb344/volumes/kubernetes.io~secret/metrics-tls major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7aad1c89-4805-4c51-a9dc-26db67c46f46/volumes/kubernetes.io~projected/kube-api-access-f6crr:{mountpoint:/var/lib/kubelet/pods/7aad1c89-4805-4c51-a9dc-26db67c46f46/volumes/kubernetes.io~projected/kube-api-access-f6crr major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87f27309-51ca-4116-b0e9-10cf9e144c36/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/87f27309-51ca-4116-b0e9-10cf9e144c36/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c10fc6a1-af69-48c2-b3dc-131e48f36b22/volumes/kubernetes.io~projected/kube-api-access-49rmt:{mountpoint:/var/lib/kubelet/pods/c10fc6a1-af69-48c2-b3dc-131e48f36b22/volumes/kubernetes.io~projected/kube-api-access-49rmt major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4f771a8-7a67-429d-8701-9c3ffa29550c/volumes/kubernetes.io~projected/kube-api-access-jzx7s:{mountpoint:/var/lib/kubelet/pods/d4f771a8-7a67-429d-8701-9c3ffa29550c/volumes/kubernetes.io~projected/kube-api-access-jzx7s major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4f771a8-7a67-429d-8701-9c3ffa29550c/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/d4f771a8-7a67-429d-8701-9c3ffa29550c/volumes/kubernetes.io~secret/signing-key major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5/volumes/kubernetes.io~projected/kube-api-access-v6vzs:{mountpoint:/var/lib/kubelet/pods/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5/volumes/kubernetes.io~projected/kube-api-access-v6vzs major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7af4112-7294-4d61-95bd-d05625f88bdc/volumes/kubernetes.io~projected/kube-api-access-9tqpc:{mountpoint:/var/lib/kubelet/pods/e7af4112-7294-4d61-95bd-d05625f88bdc/volumes/kubernetes.io~projected/kube-api-access-9tqpc major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f35c9e60-a8e9-43b2-bea6-614f6555bc03/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f35c9e60-a8e9-43b2-bea6-614f6555bc03/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f35c9e60-a8e9-43b2-bea6-614f6555bc03/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f35c9e60-a8e9-43b2-bea6-614f6555bc03/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f35c9e60-a8e9-43b2-bea6-614f6555bc03/volumes/kubernetes.io~projected/kube-api-access-vrvbt:{mountpoint:/var/lib/kubelet/pods/f35c9e60-a8e9-43b2-bea6-614f6555bc03/volumes/kubernetes.io~projected/kube-api-access-vrvbt major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9aa4593-872a-4501-9a7e-8cdeec5a92bf/volumes/kubernetes.io~projected/kube-api-access-j76cj:{mountpoint:/var/lib/kubelet/pods/f9aa4593-872a-4501-9a7e-8cdeec5a92bf/volumes/kubernetes.io~projected/kube-api-access-j76cj major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc2355fd-e26c-4d96-b399-5528863ba211/volumes/kubernetes.io~projected/kube-api-access-ct5kf:{mountpoint:/var/lib/kubelet/pods/fc2355fd-e26c-4d96-b399-5528863ba211/volumes/kubernetes.io~projected/kube-api-access-ct5kf major:0 minor:65 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/0a087a3a64200f22cfccd95f6ad5f9bec75679819376d52259ff24e112a04021/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ad4700b65d36af47b4b94b5d1f111691b7a05a11d4ef2bc86502cd21333def00/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/79625641d4ab6edfb1271eaa46568fb49447052e7ba12d82016406cb438e686c/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b2faa0a187bfee5ce9662aebfee20650aaa0d59c3399150e512502ba06685712/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/f3ba7bcedc218c0ae666344e216419bd34485eb5dd16dc2933ab723dd6526f34/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/7c51c3564439050ba17537bcdb9a1adf759efe68eb21dc0896b47c2b81abd7df/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/dd5ca6385aba00c6e65833518adc3d78711e9ad3c60cf035b4253b0f023e6dd5/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/6369029739b5a43fd5604da089a892f221fa6e9eda5770315644569db3b04073/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/8c62d0280384bbbbb9c407f0bd4c8e8c683d1fc1e31262d550f93c39b2742a6b/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/0478f25cc89f2b845dc218b9121b87809e7a5dce5d3ef515bd6de85b70161b7b/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/182469f4ecbf91c2ecd26409683f98896d4de79e56ce487806a6e7a24c36dfa0/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/c5b55904b10f6e32c43f96ac162c4d05fcfc5f9c18687c0902a1a0a974be9328/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/f0f8b33a4299f9bf57bb31326692fbbf9318f2221a31f202d4fecb76a8a7b965/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/fc1d4f32b4b7b5b96ba569dbb2f2a8e472ab5f5b98975cc896cb2b16acfc7a34/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/03a1993a696f6dfeb0db520cfed468e1bdfb5851ad414a2f0f2c3c6387257abf/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a8b80e2ecbbc8bd4c26416dc7c5d259ce2e045c16bfbe8023d4c73a46b3f287f/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/b3a43eb5ed1cab5014e209e41710b2cbf9309ef2aa76ee6513112bf5396d3fbe/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/8f6500a2345934e7aa6693facd70985d5db7caf0d8e59904a1234813731c82ee/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/7bc3754b2101ecca60c5875b52f07b98ada6782c000e4a12fc2e2ab4d27bff9c/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c3005919f183a09e894eda585e45887d218dedf8338074aa9a9015ab3ea01288/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/d05a50026a80bb8acc7c2fc9ccc9da48e8293eca49afcd0b1c37c99ace9d3f0d/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/c0f580452ec738a0134657deca8b301b2714e4de80f65d092f9da5ffb64fa0bf/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/5260d9f6c44dd67cb6a109add5b8ba5d1aa46c54b4d7e6419e180c5c208a4383/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/67161510e84725e7f1eb74840cdf0dbbb46cf53a5bc890ff8cbe59f19c20e3b4/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/6f720158f788b8a045ab8711a2f131a057b39c83df602a988000214cadade94a/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/33dafd3b84f4f3e56be2bf5734f7c2c55f8487c2952b4267502826323c499ed8/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/d70d5ed9eb9cda2d749b30bf5c210a3fb0062dbea2e4e0bb46c7813fd4cbefea/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/7deb9db7b406d649467a07a74edeb8ac172b51ec0c71f3ab65818206ba5cd49f/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/4532b7a561cddfd6093e9980145820e11827fa9c674715f0a9da1de5d3b9e222/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/41eeaaf214b33bd0c55553d0c2a5304fe167d932e8528192c21d099bd05aa19d/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/794675d8fb7c9ef4acf15011bfd58c4930a6f0d11e9c09072343089c6c09d5f9/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/0e7cb2644198eb4f6fe70c66c134c5e741f6869ba09720bbe64b0a873c34f4e9/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/24561afd46c3b15e55996d6348ba2efca4b11cd134d511387938745bc82ffa5e/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/a01bf21021284fe8aa526b05d1d398bb9d8266b3b4ab959104b9f4e100dd798b/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/3ab2bb54ddf1d0d5843efbfa1b23a60d46264ee4b903524d1be10dc0f577c5e2/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/dafc820529a06604cd0159fa3bb0e8283fd44cfd6b5d07eb499f3c9e3c2b7c56/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/cb562bfb394cde9293404b40474c3356215dea5aca178fcab731d4dbf555cd9a/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/db5d4d4f90dbfd2fc7a3033ad880e07de5d4f59bcc411730329c124f4a0bfe67/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/85eec5a89038cc9b931adf485c6c033469e93cceaacc5c48f66513657a650e22/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/0606e4f54f97741cfa691fc9355d4afedbafbb626a5c88d474786713f37aebf9/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.290638 4503 manager.go:217] Machine: {Timestamp:2026-10-30 12:51:46.289893351 +0000 UTC m=+0.113121621 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098345 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e7d27ba44568c9f4ff38fcaadefd8 SystemUUID:ec2e7d27-ba44-568c-9f4f-f38fcaadefd8 BootID:53a4ba9e-3878-4593-b9d8-22433458753e Filesystems:[{Device:/var/lib/kubelet/pods/51f1ecdc-2207-4ac7-9193-77f917cc0143/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4daea0f7cc943381817be60dcec8a8a9ae8e6ab8b33d6217107fc9094900e753/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/211fcceeb7cccbd0d573b4e75e74d6a9601b07a8b2d77d206e690a48baac2320/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1f2e503d-c234-415b-bce5-ec542e09d50b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d4f771a8-7a67-429d-8701-9c3ffa29550c/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f9d5919fe698693d8dd529bcb2b5d881abb7d72aa714248cc325c884212522d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f2e503d-c234-415b-bce5-ec542e09d50b/volumes/kubernetes.io~projected/kube-api-access-bldsp DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77484767ccf78143bd113a291395a2a537c06db202d0683795263696bdf856a4/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6e7c1377-fb2f-4b5f-83a7-b799206fb344/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:235 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e7af4112-7294-4d61-95bd-d05625f88bdc/volumes/kubernetes.io~projected/kube-api-access-9tqpc DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6011e018859e7d10b70f4ea2d1750e0b8e068413d0e24e89b6e52e7cce50f11/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a889a230f994cb2ef99626cc18f96c135e3a0b2ede65a2a71b0d07f3b0fa098/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9217d2105f8c59ba473a15b0058a619e3a3de93310651204fc5c42c6dab6dc8/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc2355fd-e26c-4d96-b399-5528863ba211/volumes/kubernetes.io~projected/kube-api-access-ct5kf DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87f27309-51ca-4116-b0e9-10cf9e144c36/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbab68617082441e7f1bef4668aec1d37f6d0dd7bf7596f51602340971802b93/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/6e7c1377-fb2f-4b5f-83a7-b799206fb344/volumes/kubernetes.io~projected/kube-api-access-zskvb DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea007a860eb40b28fe0197ca25fce4239e44bc1c76e8d212b8063fe1c90db96f/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/399d3d74faa052e4b30191ceeddfd37a932f6041b1a83399643ca8792d89b308/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c10fc6a1-af69-48c2-b3dc-131e48f36b22/volumes/kubernetes.io~projected/kube-api-access-49rmt DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30b22e49-8949-4dc1-9886-cc25b97f3436/volumes/kubernetes.io~projected/kube-api-access-2lfnw DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a493222c8d2e4ae9906d35f1657665a7128208f954db15c7718e9cb1968ae67/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f35c9e60-a8e9-43b2-bea6-614f6555bc03/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/061dda5b-09a3-493d-9c16-6d25055d5b82/volumes/kubernetes.io~projected/kube-api-access-qpmwj DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f9aa4593-872a-4501-9a7e-8cdeec5a92bf/volumes/kubernetes.io~projected/kube-api-access-j76cj DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5/volumes/kubernetes.io~projected/kube-api-access-v6vzs DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/10b4d740-fb6d-486e-b6c1-5c6115a10496/volumes/kubernetes.io~projected/kube-api-access-ndhbb DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b553c14-1755-44ed-8dec-a2d1d8259c1d/volumes/kubernetes.io~projected/kube-api-access-f7gmw DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f35c9e60-a8e9-43b2-bea6-614f6555bc03/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de47d77f14791de8fa895481fe59f9832817859f428d48da0c3c39fcb5396a92/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8acf9e7ab72f0281efb81256a559ac1c739935c6d5c72042bf94bfb50d8aecf5/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4f771a8-7a67-429d-8701-9c3ffa29550c/volumes/kubernetes.io~projected/kube-api-access-jzx7s DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52bf0edd4be66970dd5081a5a0f4864dec438a44a47223a8155d2a4541e8644f/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7aad1c89-4805-4c51-a9dc-26db67c46f46/volumes/kubernetes.io~projected/kube-api-access-f6crr DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f35c9e60-a8e9-43b2-bea6-614f6555bc03/volumes/kubernetes.io~projected/kube-api-access-vrvbt DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0207c497a7ba8391419a0401bee6de8bd19a2237c76204a21a018525023c7e7/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:399d3d74faa052e MacAddress:96:93:25:78:5c:36 Speed:10000 Mtu:8901} {Name:77484767ccf7814 MacAddress:46:66:32:c0:12:46 Speed:10000 Mtu:8901} {Name:8a493222c8d2e4a MacAddress:4e:00:ad:c2:1c:c8 Speed:10000 Mtu:8901} {Name:b0207c497a7ba83 MacAddress:c6:bb:16:30:5b:d5 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ff:97:5a:49 Speed:0 Mtu:9001} {Name:br-int MacAddress:f2:fb:78:50:51:12 Speed:0 Mtu:8901} {Name:ea007a860eb40b2 MacAddress:0a:fd:58:b6:e1:d0 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ff:97:5a:49 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:22:03:25:9c:37:d4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b2:e6:d0:f2:96:b6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.291031 4503 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.291100 4503 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.291417 4503 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.291437 4503 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-200.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.291633 4503 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.291645 4503 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.291657 4503 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.291684 4503 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.291800 4503 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.292050 4503 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.292198 4503 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.292252 4503 kubelet.go:493] "Attempting to sync node with API server" Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.292266 4503 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.292280 4503 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.292292 4503 kubelet.go:405] "Adding apiserver pod source" Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.292305 4503 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.293493 4503 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.293503 4503 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:51:46.296622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.294521 4503 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 12:51:46.305012 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.304989 4503 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 12:51:46.305012 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.305009 4503 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:51:46.305165 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.305138 4503 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 12:51:46.305450 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.305433 4503 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 12:51:46.305536 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.305475 4503 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 12:51:46.305613 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.305591 4503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 12:51:46.305613 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.305611 4503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 12:51:46.305701 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.305621 4503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 12:51:46.305701 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.305632 4503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 12:51:46.305701 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.305643 4503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 12:51:46.305701 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.305652 4503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 12:51:46.305701 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.305660 4503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 12:51:46.305701 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.305667 4503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 12:51:46.305701 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.305675 4503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 12:51:46.305701 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.305684 4503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 12:51:46.305701 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.305698 4503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 12:51:46.306169 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.305715 4503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 12:51:46.306169 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.305740 4503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 12:51:46.306169 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.305746 4503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 12:51:46.310255 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.310225 4503 server.go:1263] "Started kubelet" Oct 30 12:51:46.310455 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.310429 4503 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 12:51:46.312454 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.310685 4503 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 12:51:46.312454 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.310761 4503 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 12:51:46.312454 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.311106 4503 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 12:51:46.312454 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.311561 4503 server.go:317] "Adding debug handlers to kubelet server" Oct 30 12:51:46.310864 ip-10-0-130-200 systemd[1]: Started Kubernetes Kubelet. Oct 30 12:51:46.313166 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.313085 4503 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 12:51:46.313267 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.313187 4503 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 12:51:46.313267 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:51:46.313225 4503 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-200.ec2.internal\" not found" Oct 30 12:51:46.313397 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.313300 4503 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 12:51:46.313397 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.313311 4503 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 12:51:46.313397 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.313385 4503 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 12:51:46.313584 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.313551 4503 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:46:08 +0000 UTC" deadline="2028-06-05 15:17:30.688448188 +0000 UTC" Oct 30 12:51:46.313584 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.313582 4503 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14018h25m44.374868229s" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.314110 4503 factory.go:55] Registering systemd factory Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.314134 4503 factory.go:223] Registration of the systemd container factory successfully Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.314844 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c10fc6a1-af69-48c2-b3dc-131e48f36b22" volumeName="kubernetes.io/projected/c10fc6a1-af69-48c2-b3dc-131e48f36b22-kube-api-access-49rmt" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.314916 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5" volumeName="kubernetes.io/configmap/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-env-overrides" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.314933 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5" volumeName="kubernetes.io/secret/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.314960 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7af4112-7294-4d61-95bd-d05625f88bdc" volumeName="kubernetes.io/projected/e7af4112-7294-4d61-95bd-d05625f88bdc-kube-api-access-9tqpc" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.314972 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9aa4593-872a-4501-9a7e-8cdeec5a92bf" volumeName="kubernetes.io/projected/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-kube-api-access-j76cj" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.314984 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f35c9e60-a8e9-43b2-bea6-614f6555bc03" volumeName="kubernetes.io/empty-dir/f35c9e60-a8e9-43b2-bea6-614f6555bc03-tmp" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.314995 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e7c1377-fb2f-4b5f-83a7-b799206fb344" volumeName="kubernetes.io/projected/6e7c1377-fb2f-4b5f-83a7-b799206fb344-kube-api-access-zskvb" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315006 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87f27309-51ca-4116-b0e9-10cf9e144c36" volumeName="kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315020 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5" volumeName="kubernetes.io/projected/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-kube-api-access-v6vzs" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315032 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f2e503d-c234-415b-bce5-ec542e09d50b" volumeName="kubernetes.io/projected/1f2e503d-c234-415b-bce5-ec542e09d50b-kube-api-access-bldsp" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315043 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aad1c89-4805-4c51-a9dc-26db67c46f46" volumeName="kubernetes.io/empty-dir/7aad1c89-4805-4c51-a9dc-26db67c46f46-data-volume" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315055 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c10fc6a1-af69-48c2-b3dc-131e48f36b22" volumeName="kubernetes.io/empty-dir/c10fc6a1-af69-48c2-b3dc-131e48f36b22-tmp-dir" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315059 4503 factory.go:153] Registering CRI-O factory Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315067 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4f771a8-7a67-429d-8701-9c3ffa29550c" volumeName="kubernetes.io/projected/d4f771a8-7a67-429d-8701-9c3ffa29550c-kube-api-access-jzx7s" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315076 4503 factory.go:223] Registration of the crio container factory successfully Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315079 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5" volumeName="kubernetes.io/configmap/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-ovnkube-script-lib" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315094 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7af4112-7294-4d61-95bd-d05625f88bdc" volumeName="kubernetes.io/configmap/e7af4112-7294-4d61-95bd-d05625f88bdc-iptables-alerter-script" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315106 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc2355fd-e26c-4d96-b399-5528863ba211" volumeName="kubernetes.io/projected/fc2355fd-e26c-4d96-b399-5528863ba211-kube-api-access-ct5kf" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315137 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aad1c89-4805-4c51-a9dc-26db67c46f46" volumeName="kubernetes.io/configmap/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315151 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aad1c89-4805-4c51-a9dc-26db67c46f46" volumeName="kubernetes.io/projected/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-api-access-f6crr" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315163 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4f771a8-7a67-429d-8701-9c3ffa29550c" volumeName="kubernetes.io/configmap/d4f771a8-7a67-429d-8701-9c3ffa29550c-signing-cabundle" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315174 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9aa4593-872a-4501-9a7e-8cdeec5a92bf" volumeName="kubernetes.io/configmap/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-daemon-config" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315187 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f2e503d-c234-415b-bce5-ec542e09d50b" volumeName="kubernetes.io/secret/1f2e503d-c234-415b-bce5-ec542e09d50b-cert" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315199 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30b22e49-8949-4dc1-9886-cc25b97f3436" volumeName="kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315211 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e7c1377-fb2f-4b5f-83a7-b799206fb344" volumeName="kubernetes.io/configmap/6e7c1377-fb2f-4b5f-83a7-b799206fb344-config-volume" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315224 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f35c9e60-a8e9-43b2-bea6-614f6555bc03" volumeName="kubernetes.io/empty-dir/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-tuned" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315237 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9aa4593-872a-4501-9a7e-8cdeec5a92bf" volumeName="kubernetes.io/configmap/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-cni-binary-copy" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315250 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc2355fd-e26c-4d96-b399-5528863ba211" volumeName="kubernetes.io/configmap/fc2355fd-e26c-4d96-b399-5528863ba211-cni-binary-copy" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315263 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e7c1377-fb2f-4b5f-83a7-b799206fb344" volumeName="kubernetes.io/empty-dir/6e7c1377-fb2f-4b5f-83a7-b799206fb344-tmp-dir" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315276 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e7c1377-fb2f-4b5f-83a7-b799206fb344" volumeName="kubernetes.io/secret/6e7c1377-fb2f-4b5f-83a7-b799206fb344-metrics-tls" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315285 4503 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315297 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f35c9e60-a8e9-43b2-bea6-614f6555bc03" volumeName="kubernetes.io/projected/f35c9e60-a8e9-43b2-bea6-614f6555bc03-kube-api-access-vrvbt" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315309 4503 factory.go:103] Registering Raw factory Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315329 4503 manager.go:1199] Started watching for new ooms in manager Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315310 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc2355fd-e26c-4d96-b399-5528863ba211" volumeName="kubernetes.io/configmap/fc2355fd-e26c-4d96-b399-5528863ba211-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315389 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10b4d740-fb6d-486e-b6c1-5c6115a10496" volumeName="kubernetes.io/projected/10b4d740-fb6d-486e-b6c1-5c6115a10496-kube-api-access-ndhbb" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315412 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51f1ecdc-2207-4ac7-9193-77f917cc0143" volumeName="kubernetes.io/configmap/51f1ecdc-2207-4ac7-9193-77f917cc0143-konnectivity-ca" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315426 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b553c14-1755-44ed-8dec-a2d1d8259c1d" volumeName="kubernetes.io/projected/6b553c14-1755-44ed-8dec-a2d1d8259c1d-kube-api-access-f7gmw" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315439 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4f771a8-7a67-429d-8701-9c3ffa29550c" volumeName="kubernetes.io/secret/d4f771a8-7a67-429d-8701-9c3ffa29550c-signing-key" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315450 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5" volumeName="kubernetes.io/configmap/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-ovnkube-config" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315465 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc2355fd-e26c-4d96-b399-5528863ba211" volumeName="kubernetes.io/configmap/fc2355fd-e26c-4d96-b399-5528863ba211-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315478 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="061dda5b-09a3-493d-9c16-6d25055d5b82" volumeName="kubernetes.io/projected/061dda5b-09a3-493d-9c16-6d25055d5b82-kube-api-access-qpmwj" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315491 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10b4d740-fb6d-486e-b6c1-5c6115a10496" volumeName="kubernetes.io/configmap/10b4d740-fb6d-486e-b6c1-5c6115a10496-serviceca" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315502 4503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51f1ecdc-2207-4ac7-9193-77f917cc0143" volumeName="kubernetes.io/secret/51f1ecdc-2207-4ac7-9193-77f917cc0143-agent-certs" seLinuxMountContext="" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315520 4503 reconstruct.go:97] "Volume reconstruction finished" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315528 4503 reconciler.go:29] "Reconciler: start to sync state" Oct 30 12:51:46.314907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.315731 4503 manager.go:319] Starting recovery of all containers Oct 30 12:51:46.352311 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.352259 4503 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 12:51:46.354798 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.353933 4503 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 12:51:46.354798 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.353970 4503 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 12:51:46.354798 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.353993 4503 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 12:51:46.354798 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:51:46.354038 4503 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 12:51:46.357183 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.357159 4503 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 12:51:46.373285 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.373213 4503 generic.go:356] "Generic (PLEG): container finished" podID="ba36f1221f97bdfc5568490402fa51dc" containerID="75a3fe2043c62b86c18ffab41e87a824dc1f41da973c243ac0eee7b26d5e2400" exitCode=0 Oct 30 12:51:46.374815 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.374786 4503 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:46.375303 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.375278 4503 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:46.375601 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.375580 4503 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:46.380043 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.380020 4503 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:46.388530 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.388502 4503 generic.go:356] "Generic (PLEG): container finished" podID="fc2355fd-e26c-4d96-b399-5528863ba211" containerID="525246909e3c978f648cd09eda8fa7b9553615436497eb6816bacb3695084d87" exitCode=0 Oct 30 12:51:46.388530 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.388528 4503 generic.go:356] "Generic (PLEG): container finished" podID="fc2355fd-e26c-4d96-b399-5528863ba211" containerID="fbe92d0f15191521b9f2642ac5822b0f80c2cf2174dc4cd0dfd961dd3f2bbd06" exitCode=0 Oct 30 12:51:46.388530 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.388535 4503 generic.go:356] "Generic (PLEG): container finished" podID="fc2355fd-e26c-4d96-b399-5528863ba211" containerID="58f583c5ba8c93eb6047fd0f6ff15a27961482021a988dac5396945f454163c9" exitCode=0 Oct 30 12:51:46.388691 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.388541 4503 generic.go:356] "Generic (PLEG): container finished" podID="fc2355fd-e26c-4d96-b399-5528863ba211" containerID="49fc33c41e70a7ff666b4f86f335be08a970cfdc41556106cd071bb3d6b4bb3d" exitCode=0 Oct 30 12:51:46.388691 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.388546 4503 generic.go:356] "Generic (PLEG): container finished" podID="fc2355fd-e26c-4d96-b399-5528863ba211" containerID="22bfdd65ed8537f2640e76c783e7309025712cb82d315ffd4bb742f485587e89" exitCode=0 Oct 30 12:51:46.388691 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.388551 4503 generic.go:356] "Generic (PLEG): container finished" podID="fc2355fd-e26c-4d96-b399-5528863ba211" containerID="677a730fe90161fd5e7af9630de35d05c50f5bcf96de08ea490447a8787e7cca" exitCode=0 Oct 30 12:51:46.393036 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.393016 4503 manager.go:324] Recovery completed Oct 30 12:51:46.407325 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.407305 4503 cpu_manager.go:226] "Starting" policy="none" Oct 30 12:51:46.407325 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.407318 4503 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 12:51:46.407466 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.407334 4503 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 12:51:46.407466 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.407445 4503 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 12:51:46.407529 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.407456 4503 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 12:51:46.407529 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.407478 4503 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 12:51:46.407529 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.407484 4503 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 12:51:46.407529 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.407489 4503 policy_none.go:50] "Start" Oct 30 12:51:46.407529 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.407497 4503 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 12:51:46.407529 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.407506 4503 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 12:51:46.407688 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.407636 4503 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 12:51:46.407688 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.407643 4503 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 12:51:46.407688 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.407648 4503 policy_none.go:44] "Start" Oct 30 12:51:46.412085 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.411564 4503 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 12:51:46.417071 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.417057 4503 manager.go:341] "Starting Device Plugin manager" Oct 30 12:51:46.417151 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:51:46.417092 4503 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 12:51:46.417151 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.417103 4503 server.go:87] "Starting device plugin registration server" Oct 30 12:51:46.417422 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.417405 4503 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 12:51:46.417528 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.417420 4503 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 12:51:46.417585 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.417526 4503 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 12:51:46.417646 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.417612 4503 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 12:51:46.417717 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.417649 4503 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 12:51:46.417717 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.417656 4503 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 12:51:46.417810 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.417694 4503 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:51:46.417639385Z","UUID":"513292be-ad07-4e68-9267-7a14fa96db81","Handler":null,"Name":"","Endpoint":""} Oct 30 12:51:46.418825 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:51:46.418805 4503 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 12:51:46.419244 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.419228 4503 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 12:51:46.419333 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.419259 4503 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 12:51:46.454382 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.454336 4503 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal"] Oct 30 12:51:46.454830 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.454803 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" event={"ID":"ba36f1221f97bdfc5568490402fa51dc","Type":"ContainerStarted","Data":"5e6e360e8e218637ae620654081571c16cd743d47fa475d84648ace53570f88f"} Oct 30 12:51:46.454870 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.454838 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" event={"ID":"ba36f1221f97bdfc5568490402fa51dc","Type":"ContainerDied","Data":"75a3fe2043c62b86c18ffab41e87a824dc1f41da973c243ac0eee7b26d5e2400"} Oct 30 12:51:46.454870 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.454854 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" event={"ID":"ba36f1221f97bdfc5568490402fa51dc","Type":"ContainerStarted","Data":"52bf0edd4be66970dd5081a5a0f4864dec438a44a47223a8155d2a4541e8644f"} Oct 30 12:51:46.454925 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.454900 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal" event={"ID":"f1dd3ede18217038ba719ab8e2dbe78f","Type":"ContainerStarted","Data":"194151d7c5b174859a6ed93bfd485e919d7586e01020f339c8dbffb3587cdedc"} Oct 30 12:51:46.454925 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.454911 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal" event={"ID":"f1dd3ede18217038ba719ab8e2dbe78f","Type":"ContainerStarted","Data":"f9217d2105f8c59ba473a15b0058a619e3a3de93310651204fc5c42c6dab6dc8"} Oct 30 12:51:46.455027 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.455012 4503 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal" Oct 30 12:51:46.455064 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.455040 4503 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" Oct 30 12:51:46.477384 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.477360 4503 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:51:46.477485 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:51:46.477413 4503 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-200.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal" Oct 30 12:51:46.478600 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.478577 4503 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:51:46.478710 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:51:46.478618 4503 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" Oct 30 12:51:46.517636 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.517607 4503 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:46.519098 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.519076 4503 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:46.519198 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.519115 4503 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:46.519198 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.519129 4503 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:46.519198 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.519162 4503 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-200.ec2.internal" Oct 30 12:51:46.558358 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.558328 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba36f1221f97bdfc5568490402fa51dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal\" (UID: \"ba36f1221f97bdfc5568490402fa51dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" Oct 30 12:51:46.558472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.558360 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba36f1221f97bdfc5568490402fa51dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal\" (UID: \"ba36f1221f97bdfc5568490402fa51dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" Oct 30 12:51:46.558472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.558384 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1dd3ede18217038ba719ab8e2dbe78f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-200.ec2.internal\" (UID: \"f1dd3ede18217038ba719ab8e2dbe78f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal" Oct 30 12:51:46.558553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.558474 4503 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-200.ec2.internal" Oct 30 12:51:46.558553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.558542 4503 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-200.ec2.internal" Oct 30 12:51:46.659463 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.659426 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba36f1221f97bdfc5568490402fa51dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal\" (UID: \"ba36f1221f97bdfc5568490402fa51dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" Oct 30 12:51:46.659644 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.659489 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba36f1221f97bdfc5568490402fa51dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal\" (UID: \"ba36f1221f97bdfc5568490402fa51dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" Oct 30 12:51:46.659644 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.659516 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1dd3ede18217038ba719ab8e2dbe78f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-200.ec2.internal\" (UID: \"f1dd3ede18217038ba719ab8e2dbe78f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal" Oct 30 12:51:46.659644 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.659443 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba36f1221f97bdfc5568490402fa51dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal\" (UID: \"ba36f1221f97bdfc5568490402fa51dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" Oct 30 12:51:46.659644 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.659563 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1dd3ede18217038ba719ab8e2dbe78f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-200.ec2.internal\" (UID: \"f1dd3ede18217038ba719ab8e2dbe78f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal" Oct 30 12:51:46.659644 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:46.659572 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba36f1221f97bdfc5568490402fa51dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal\" (UID: \"ba36f1221f97bdfc5568490402fa51dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" Oct 30 12:51:47.293408 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.293375 4503 apiserver.go:52] "Watching apiserver" Oct 30 12:51:47.338891 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.338863 4503 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 12:51:47.339323 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.339305 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-f4rcm","kube-system/global-pull-secret-syncer-r5lrn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49","openshift-insights/insights-runtime-extractor-hbsfb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal","openshift-multus/multus-additional-cni-plugins-cl4bq","kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal","openshift-dns/node-resolver-z6m2l","openshift-multus/multus-8l62c","openshift-image-registry/node-ca-kvsqc","openshift-multus/network-metrics-daemon-7l5g4","openshift-network-diagnostics/network-check-target-xb4w4","kube-system/konnectivity-agent-pz66k","openshift-ovn-kubernetes/ovnkube-node-k2v62","openshift-dns/dns-default-b7kx7","openshift-network-operator/iptables-alerter-4rj8r","openshift-cluster-node-tuning-operator/tuned-z9f5d","openshift-ingress-canary/ingress-canary-p8vd8"] Oct 30 12:51:47.340041 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.340019 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:47.348023 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.348001 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:47.400177 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.400151 4503 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal" Oct 30 12:51:47.400312 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.400267 4503 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" Oct 30 12:51:47.401107 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.401086 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 12:51:47.401107 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.401104 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 12:51:47.401246 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.401096 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 12:51:47.401246 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.401151 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 12:51:47.401246 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.401174 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 12:51:47.401246 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.401191 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 12:51:47.401246 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.401199 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 12:51:47.401246 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.401229 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 12:51:47.401444 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.401285 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 12:51:47.401444 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.401381 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:47.402083 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.402057 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 12:51:47.402272 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.402257 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 12:51:47.402353 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.402340 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 12:51:47.402791 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.402770 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 12:51:47.402893 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.402851 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7j54b\"" Oct 30 12:51:47.402893 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.402866 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wqgtx\"" Oct 30 12:51:47.402893 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.402875 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xj7mb\"" Oct 30 12:51:47.402893 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.402871 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-68fxr\"" Oct 30 12:51:47.403120 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.402904 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 12:51:47.403120 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.402999 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-45hdr\"" Oct 30 12:51:47.403120 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.403002 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-sfhnn\"" Oct 30 12:51:47.403120 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.403081 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hszmd\"" Oct 30 12:51:47.403120 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.403100 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 12:51:47.403120 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.403110 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-87w8v\"" Oct 30 12:51:47.403400 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.403168 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qjjfj\"" Oct 30 12:51:47.403400 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.403187 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 12:51:47.403400 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.403225 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 12:51:47.403400 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.403230 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m2lrf\"" Oct 30 12:51:47.403400 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.403293 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9ppdm\"" Oct 30 12:51:47.403400 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.403375 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-v4xk8\"" Oct 30 12:51:47.403675 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.403452 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ksd25\"" Oct 30 12:51:47.403675 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.403565 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ktwvb\"" Oct 30 12:51:47.403675 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.403652 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-48745\"" Oct 30 12:51:47.403826 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.403720 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 12:51:47.403826 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.403818 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bhhxq\"" Oct 30 12:51:47.411142 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.411121 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 12:51:47.411142 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.411137 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 12:51:47.411296 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.411143 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 12:51:47.411296 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.411207 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 12:51:47.411296 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.411237 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:47.411296 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.411260 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 12:51:47.411296 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.411271 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 12:51:47.411296 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.411285 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 12:51:47.414057 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.414034 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:47.414160 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.414131 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:47.414367 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.414340 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 12:51:47.414429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.414347 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 12:51:47.414478 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.414457 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 12:51:47.414556 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.414514 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 12:51:47.414631 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.414616 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 12:51:47.414760 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.414652 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 12:51:47.414760 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.414713 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 12:51:47.414862 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.414776 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 12:51:47.414862 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.414834 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 12:51:47.414862 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.414840 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 12:51:47.415041 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.414843 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 12:51:47.415041 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.414875 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 12:51:47.415041 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.415039 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 12:51:47.415148 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.415133 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 12:51:47.415148 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.415142 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 12:51:47.415305 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.415293 4503 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 12:51:47.425604 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.425584 4503 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:51:47.425711 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:51:47.425630 4503 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-200.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-200.ec2.internal" Oct 30 12:51:47.425711 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.425654 4503 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:51:47.425711 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:51:47.425694 4503 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-200.ec2.internal" Oct 30 12:51:47.464235 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464199 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6crr\" (UniqueName: \"kubernetes.io/projected/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-api-access-f6crr\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:47.464235 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464233 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7af4112-7294-4d61-95bd-d05625f88bdc-host-slash\") pod \"iptables-alerter-4rj8r\" (UID: \"e7af4112-7294-4d61-95bd-d05625f88bdc\") " pod="openshift-network-operator/iptables-alerter-4rj8r" Oct 30 12:51:47.464530 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464252 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-slash\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.464530 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464270 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e7c1377-fb2f-4b5f-83a7-b799206fb344-tmp-dir\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:47.464530 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464287 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49rmt\" (UniqueName: \"kubernetes.io/projected/c10fc6a1-af69-48c2-b3dc-131e48f36b22-kube-api-access-49rmt\") pod \"node-resolver-z6m2l\" (UID: \"c10fc6a1-af69-48c2-b3dc-131e48f36b22\") " pod="openshift-dns/node-resolver-z6m2l" Oct 30 12:51:47.464530 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464350 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-system-cni-dir\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.464530 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464400 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-run-netns\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.464530 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464427 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.464530 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464446 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-cni-netd\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.464530 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464482 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.464530 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464505 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e7c1377-fb2f-4b5f-83a7-b799206fb344-metrics-tls\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:47.464530 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464527 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-socket-dir-parent\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.465005 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464661 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-etc-kubernetes\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.465005 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464714 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:47.465005 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464716 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e7c1377-fb2f-4b5f-83a7-b799206fb344-tmp-dir\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:47.465005 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464758 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e7c1377-fb2f-4b5f-83a7-b799206fb344-metrics-tls\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:47.465005 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464819 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-sysconfig\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.465005 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464838 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-kubernetes\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.465005 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464856 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrvbt\" (UniqueName: \"kubernetes.io/projected/f35c9e60-a8e9-43b2-bea6-614f6555bc03-kube-api-access-vrvbt\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.465005 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464885 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc2355fd-e26c-4d96-b399-5528863ba211-cnibin\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:47.465005 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464988 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-ovnkube-config\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.465005 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.464998 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465024 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51f1ecdc-2207-4ac7-9193-77f917cc0143-konnectivity-ca\") pod \"konnectivity-agent-pz66k\" (UID: \"51f1ecdc-2207-4ac7-9193-77f917cc0143\") " pod="kube-system/konnectivity-agent-pz66k" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465057 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87f27309-51ca-4116-b0e9-10cf9e144c36-kubelet-config\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465082 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465110 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc2355fd-e26c-4d96-b399-5528863ba211-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465129 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-etc-openvswitch\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465144 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-run-ovn\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465161 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465177 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10b4d740-fb6d-486e-b6c1-5c6115a10496-serviceca\") pod \"node-ca-kvsqc\" (UID: \"10b4d740-fb6d-486e-b6c1-5c6115a10496\") " pod="openshift-image-registry/node-ca-kvsqc" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465239 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndhbb\" (UniqueName: \"kubernetes.io/projected/10b4d740-fb6d-486e-b6c1-5c6115a10496-kube-api-access-ndhbb\") pod \"node-ca-kvsqc\" (UID: \"10b4d740-fb6d-486e-b6c1-5c6115a10496\") " pod="openshift-image-registry/node-ca-kvsqc" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465247 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-ovnkube-config\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465290 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f2e503d-c234-415b-bce5-ec542e09d50b-cert\") pod \"ingress-canary-p8vd8\" (UID: \"1f2e503d-c234-415b-bce5-ec542e09d50b\") " pod="openshift-ingress-canary/ingress-canary-p8vd8" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465318 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7aad1c89-4805-4c51-a9dc-26db67c46f46-data-volume\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465343 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-systemd\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465362 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10b4d740-fb6d-486e-b6c1-5c6115a10496-serviceca\") pod \"node-ca-kvsqc\" (UID: \"10b4d740-fb6d-486e-b6c1-5c6115a10496\") " pod="openshift-image-registry/node-ca-kvsqc" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465370 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-ovn-node-metrics-cert\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465393 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87f27309-51ca-4116-b0e9-10cf9e144c36-original-pull-secret\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465403 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51f1ecdc-2207-4ac7-9193-77f917cc0143-konnectivity-ca\") pod \"konnectivity-agent-pz66k\" (UID: \"51f1ecdc-2207-4ac7-9193-77f917cc0143\") " pod="kube-system/konnectivity-agent-pz66k" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465395 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51f1ecdc-2207-4ac7-9193-77f917cc0143-agent-certs\") pod \"konnectivity-agent-pz66k\" (UID: \"51f1ecdc-2207-4ac7-9193-77f917cc0143\") " pod="kube-system/konnectivity-agent-pz66k" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465416 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7aad1c89-4805-4c51-a9dc-26db67c46f46-data-volume\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465446 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d4f771a8-7a67-429d-8701-9c3ffa29550c-signing-cabundle\") pod \"service-ca-5fffc97bdf-f4rcm\" (UID: \"d4f771a8-7a67-429d-8701-9c3ffa29550c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" Oct 30 12:51:47.465472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465475 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-device-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465511 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-sysctl-conf\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465533 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-ovn-node-metrics-cert\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465534 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-sys\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465567 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-host\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465577 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f2e503d-c234-415b-bce5-ec542e09d50b-cert\") pod \"ingress-canary-p8vd8\" (UID: \"1f2e503d-c234-415b-bce5-ec542e09d50b\") " pod="openshift-ingress-canary/ingress-canary-p8vd8" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465592 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-systemd-units\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465618 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-cni-bin\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465619 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d4f771a8-7a67-429d-8701-9c3ffa29550c-signing-cabundle\") pod \"service-ca-5fffc97bdf-f4rcm\" (UID: \"d4f771a8-7a67-429d-8701-9c3ffa29550c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465639 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51f1ecdc-2207-4ac7-9193-77f917cc0143-agent-certs\") pod \"konnectivity-agent-pz66k\" (UID: \"51f1ecdc-2207-4ac7-9193-77f917cc0143\") " pod="kube-system/konnectivity-agent-pz66k" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465659 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfnw\" (UniqueName: \"kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw\") pod \"network-check-target-xb4w4\" (UID: \"30b22e49-8949-4dc1-9886-cc25b97f3436\") " pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465779 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7c1377-fb2f-4b5f-83a7-b799206fb344-config-volume\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465815 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct5kf\" (UniqueName: \"kubernetes.io/projected/fc2355fd-e26c-4d96-b399-5528863ba211-kube-api-access-ct5kf\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465920 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-node-log\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465925 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7c1377-fb2f-4b5f-83a7-b799206fb344-config-volume\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.465973 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zskvb\" (UniqueName: \"kubernetes.io/projected/6e7c1377-fb2f-4b5f-83a7-b799206fb344-kube-api-access-zskvb\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466001 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bldsp\" (UniqueName: \"kubernetes.io/projected/1f2e503d-c234-415b-bce5-ec542e09d50b-kube-api-access-bldsp\") pod \"ingress-canary-p8vd8\" (UID: \"1f2e503d-c234-415b-bce5-ec542e09d50b\") " pod="openshift-ingress-canary/ingress-canary-p8vd8" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466024 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-run-netns\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466103 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-conf-dir\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466148 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-daemon-config\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466199 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j76cj\" (UniqueName: \"kubernetes.io/projected/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-kube-api-access-j76cj\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466252 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7gmw\" (UniqueName: \"kubernetes.io/projected/6b553c14-1755-44ed-8dec-a2d1d8259c1d-kube-api-access-f7gmw\") pod \"network-metrics-daemon-7l5g4\" (UID: \"6b553c14-1755-44ed-8dec-a2d1d8259c1d\") " pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466277 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc2355fd-e26c-4d96-b399-5528863ba211-cni-binary-copy\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466293 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc2355fd-e26c-4d96-b399-5528863ba211-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466317 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-kubelet\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466341 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-run-openvswitch\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466380 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-cni-binary-copy\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466393 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-daemon-config\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466411 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-run-systemd\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.466429 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466436 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vzs\" (UniqueName: \"kubernetes.io/projected/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-kube-api-access-v6vzs\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466476 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c10fc6a1-af69-48c2-b3dc-131e48f36b22-tmp-dir\") pod \"node-resolver-z6m2l\" (UID: \"c10fc6a1-af69-48c2-b3dc-131e48f36b22\") " pod="openshift-dns/node-resolver-z6m2l" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466509 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs\") pod \"network-metrics-daemon-7l5g4\" (UID: \"6b553c14-1755-44ed-8dec-a2d1d8259c1d\") " pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466528 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc2355fd-e26c-4d96-b399-5528863ba211-cni-binary-copy\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466557 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-modprobe-d\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466608 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c10fc6a1-af69-48c2-b3dc-131e48f36b22-tmp-dir\") pod \"node-resolver-z6m2l\" (UID: \"c10fc6a1-af69-48c2-b3dc-131e48f36b22\") " pod="openshift-dns/node-resolver-z6m2l" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466614 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d4f771a8-7a67-429d-8701-9c3ffa29550c-signing-key\") pod \"service-ca-5fffc97bdf-f4rcm\" (UID: \"d4f771a8-7a67-429d-8701-9c3ffa29550c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466626 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-cni-binary-copy\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466629 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc2355fd-e26c-4d96-b399-5528863ba211-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466651 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-cnibin\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466694 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-var-lib-cni-multus\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466721 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-ovnkube-script-lib\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466741 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzx7s\" (UniqueName: \"kubernetes.io/projected/d4f771a8-7a67-429d-8701-9c3ffa29550c-kube-api-access-jzx7s\") pod \"service-ca-5fffc97bdf-f4rcm\" (UID: \"d4f771a8-7a67-429d-8701-9c3ffa29550c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466759 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-run-k8s-cni-cncf-io\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466775 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-hostroot\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466789 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-registration-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466838 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpmwj\" (UniqueName: \"kubernetes.io/projected/061dda5b-09a3-493d-9c16-6d25055d5b82-kube-api-access-qpmwj\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.466863 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d4f771a8-7a67-429d-8701-9c3ffa29550c-signing-key\") pod \"service-ca-5fffc97bdf-f4rcm\" (UID: \"d4f771a8-7a67-429d-8701-9c3ffa29550c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467281 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e7af4112-7294-4d61-95bd-d05625f88bdc-iptables-alerter-script\") pod \"iptables-alerter-4rj8r\" (UID: \"e7af4112-7294-4d61-95bd-d05625f88bdc\") " pod="openshift-network-operator/iptables-alerter-4rj8r" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467385 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-var-lib-cni-bin\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467424 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-var-lib-kubelet\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467467 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.467553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467540 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc2355fd-e26c-4d96-b399-5528863ba211-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467590 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-var-lib-openvswitch\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467621 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87f27309-51ca-4116-b0e9-10cf9e144c36-dbus\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467653 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c10fc6a1-af69-48c2-b3dc-131e48f36b22-hosts-file\") pod \"node-resolver-z6m2l\" (UID: \"c10fc6a1-af69-48c2-b3dc-131e48f36b22\") " pod="openshift-dns/node-resolver-z6m2l" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467684 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-cni-dir\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467708 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-etc-selinux\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467739 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f35c9e60-a8e9-43b2-bea6-614f6555bc03-tmp\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467770 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tqpc\" (UniqueName: \"kubernetes.io/projected/e7af4112-7294-4d61-95bd-d05625f88bdc-kube-api-access-9tqpc\") pod \"iptables-alerter-4rj8r\" (UID: \"e7af4112-7294-4d61-95bd-d05625f88bdc\") " pod="openshift-network-operator/iptables-alerter-4rj8r" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467795 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc2355fd-e26c-4d96-b399-5528863ba211-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467798 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-log-socket\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467883 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-env-overrides\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467880 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-ovnkube-script-lib\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467914 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-run\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467958 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e7af4112-7294-4d61-95bd-d05625f88bdc-iptables-alerter-script\") pod \"iptables-alerter-4rj8r\" (UID: \"e7af4112-7294-4d61-95bd-d05625f88bdc\") " pod="openshift-network-operator/iptables-alerter-4rj8r" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467986 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-var-lib-kubelet\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.467999 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f35c9e60-a8e9-43b2-bea6-614f6555bc03-tmp\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.468021 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-run-multus-certs\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.468075 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7aad1c89-4805-4c51-a9dc-26db67c46f46-crio-socket\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.468141 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-socket-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.468178 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-sys-fs\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.468191 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-env-overrides\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.468227 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-sysctl-d\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.468277 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-lib-modules\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.468316 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-tuned\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.468355 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc2355fd-e26c-4d96-b399-5528863ba211-system-cni-dir\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.468433 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10b4d740-fb6d-486e-b6c1-5c6115a10496-host\") pod \"node-ca-kvsqc\" (UID: \"10b4d740-fb6d-486e-b6c1-5c6115a10496\") " pod="openshift-image-registry/node-ca-kvsqc" Oct 30 12:51:47.468537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.468441 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-tuned\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.496121 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.496091 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6crr\" (UniqueName: \"kubernetes.io/projected/7aad1c89-4805-4c51-a9dc-26db67c46f46-kube-api-access-f6crr\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:47.498444 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.498420 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndhbb\" (UniqueName: \"kubernetes.io/projected/10b4d740-fb6d-486e-b6c1-5c6115a10496-kube-api-access-ndhbb\") pod \"node-ca-kvsqc\" (UID: \"10b4d740-fb6d-486e-b6c1-5c6115a10496\") " pod="openshift-image-registry/node-ca-kvsqc" Oct 30 12:51:47.499141 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.499091 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49rmt\" (UniqueName: \"kubernetes.io/projected/c10fc6a1-af69-48c2-b3dc-131e48f36b22-kube-api-access-49rmt\") pod \"node-resolver-z6m2l\" (UID: \"c10fc6a1-af69-48c2-b3dc-131e48f36b22\") " pod="openshift-dns/node-resolver-z6m2l" Oct 30 12:51:47.499283 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.499262 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j76cj\" (UniqueName: \"kubernetes.io/projected/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-kube-api-access-j76cj\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.499283 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.499274 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzx7s\" (UniqueName: \"kubernetes.io/projected/d4f771a8-7a67-429d-8701-9c3ffa29550c-kube-api-access-jzx7s\") pod \"service-ca-5fffc97bdf-f4rcm\" (UID: \"d4f771a8-7a67-429d-8701-9c3ffa29550c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f4rcm" Oct 30 12:51:47.499283 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.499283 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tqpc\" (UniqueName: \"kubernetes.io/projected/e7af4112-7294-4d61-95bd-d05625f88bdc-kube-api-access-9tqpc\") pod \"iptables-alerter-4rj8r\" (UID: \"e7af4112-7294-4d61-95bd-d05625f88bdc\") " pod="openshift-network-operator/iptables-alerter-4rj8r" Oct 30 12:51:47.499476 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.499315 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct5kf\" (UniqueName: \"kubernetes.io/projected/fc2355fd-e26c-4d96-b399-5528863ba211-kube-api-access-ct5kf\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:47.499476 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.499333 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bldsp\" (UniqueName: \"kubernetes.io/projected/1f2e503d-c234-415b-bce5-ec542e09d50b-kube-api-access-bldsp\") pod \"ingress-canary-p8vd8\" (UID: \"1f2e503d-c234-415b-bce5-ec542e09d50b\") " pod="openshift-ingress-canary/ingress-canary-p8vd8" Oct 30 12:51:47.499476 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.499390 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lfnw\" (UniqueName: \"kubernetes.io/projected/30b22e49-8949-4dc1-9886-cc25b97f3436-kube-api-access-2lfnw\") pod \"network-check-target-xb4w4\" (UID: \"30b22e49-8949-4dc1-9886-cc25b97f3436\") " pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:47.499636 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.499615 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7gmw\" (UniqueName: \"kubernetes.io/projected/6b553c14-1755-44ed-8dec-a2d1d8259c1d-kube-api-access-f7gmw\") pod \"network-metrics-daemon-7l5g4\" (UID: \"6b553c14-1755-44ed-8dec-a2d1d8259c1d\") " pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:47.499830 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.499806 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zskvb\" (UniqueName: \"kubernetes.io/projected/6e7c1377-fb2f-4b5f-83a7-b799206fb344-kube-api-access-zskvb\") pod \"dns-default-b7kx7\" (UID: \"6e7c1377-fb2f-4b5f-83a7-b799206fb344\") " pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:47.499889 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.499833 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpmwj\" (UniqueName: \"kubernetes.io/projected/061dda5b-09a3-493d-9c16-6d25055d5b82-kube-api-access-qpmwj\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.499889 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.499855 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrvbt\" (UniqueName: \"kubernetes.io/projected/f35c9e60-a8e9-43b2-bea6-614f6555bc03-kube-api-access-vrvbt\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.500682 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.500662 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6vzs\" (UniqueName: \"kubernetes.io/projected/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-kube-api-access-v6vzs\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.569078 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.568972 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-kubernetes\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.569078 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569017 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc2355fd-e26c-4d96-b399-5528863ba211-cnibin\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:47.569078 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569039 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87f27309-51ca-4116-b0e9-10cf9e144c36-kubelet-config\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:47.569078 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569062 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:47.569346 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569087 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc2355fd-e26c-4d96-b399-5528863ba211-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:47.569346 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569101 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc2355fd-e26c-4d96-b399-5528863ba211-cnibin\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:47.569346 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569122 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87f27309-51ca-4116-b0e9-10cf9e144c36-kubelet-config\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:47.569346 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569112 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-etc-openvswitch\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.569346 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569101 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-kubernetes\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.569346 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569168 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-run-ovn\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.569346 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569175 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc2355fd-e26c-4d96-b399-5528863ba211-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:47.569346 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569194 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-etc-openvswitch\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.569346 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:51:47.569206 4503 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:51:47.569346 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569235 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-run-ovn\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.569346 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569273 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-systemd\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.569346 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569307 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-systemd\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.569346 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:51:47.569334 4503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls podName:7aad1c89-4805-4c51-a9dc-26db67c46f46 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:48.069301535 +0000 UTC m=+1.892529806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbsfb" (UID: "7aad1c89-4805-4c51-a9dc-26db67c46f46") : secret "insights-runtime-extractor-tls" not found Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569367 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-device-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569384 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-sysctl-conf\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569399 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-sys\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569412 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-host\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569435 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-systemd-units\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569445 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-device-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569473 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-sys\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569502 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-host\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569509 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-systemd-units\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569532 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-sysctl-conf\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569531 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-cni-bin\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569571 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-node-log\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569588 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-run-netns\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569607 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-conf-dir\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569635 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-run-netns\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569642 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-node-log\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569636 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-kubelet\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569664 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-cni-bin\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569647 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-conf-dir\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569667 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-kubelet\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569679 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-run-openvswitch\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569701 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-run-systemd\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569720 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs\") pod \"network-metrics-daemon-7l5g4\" (UID: \"6b553c14-1755-44ed-8dec-a2d1d8259c1d\") " pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569744 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-modprobe-d\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569752 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-run-openvswitch\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569772 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-cnibin\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.569780 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569787 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-var-lib-cni-multus\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569811 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-run-k8s-cni-cncf-io\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569835 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-hostroot\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569852 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-cnibin\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569856 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-var-lib-cni-multus\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569862 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-run-systemd\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569858 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-registration-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569892 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-modprobe-d\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569902 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-hostroot\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569908 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-registration-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569914 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-var-lib-cni-bin\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569920 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-run-k8s-cni-cncf-io\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569960 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-var-lib-kubelet\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569967 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-var-lib-cni-bin\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.569985 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570004 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-var-lib-kubelet\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570007 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-var-lib-openvswitch\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570028 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87f27309-51ca-4116-b0e9-10cf9e144c36-dbus\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570056 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-var-lib-openvswitch\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570071 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570075 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c10fc6a1-af69-48c2-b3dc-131e48f36b22-hosts-file\") pod \"node-resolver-z6m2l\" (UID: \"c10fc6a1-af69-48c2-b3dc-131e48f36b22\") " pod="openshift-dns/node-resolver-z6m2l" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570086 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87f27309-51ca-4116-b0e9-10cf9e144c36-dbus\") pod \"global-pull-secret-syncer-r5lrn\" (UID: \"87f27309-51ca-4116-b0e9-10cf9e144c36\") " pod="kube-system/global-pull-secret-syncer-r5lrn" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570107 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c10fc6a1-af69-48c2-b3dc-131e48f36b22-hosts-file\") pod \"node-resolver-z6m2l\" (UID: \"c10fc6a1-af69-48c2-b3dc-131e48f36b22\") " pod="openshift-dns/node-resolver-z6m2l" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570112 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-cni-dir\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570136 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-etc-selinux\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570157 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-cni-dir\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570161 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-log-socket\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570186 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-run\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570190 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-log-socket\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570202 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-etc-selinux\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570209 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-var-lib-kubelet\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570233 4503 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570248 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-run-multus-certs\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570258 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-run\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570260 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-var-lib-kubelet\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570274 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7aad1c89-4805-4c51-a9dc-26db67c46f46-crio-socket\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570274 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-host-run-multus-certs\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570291 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-socket-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570305 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-sys-fs\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570310 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7aad1c89-4805-4c51-a9dc-26db67c46f46-crio-socket\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570318 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-sysctl-d\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570344 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-lib-modules\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570369 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-sys-fs\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570370 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc2355fd-e26c-4d96-b399-5528863ba211-system-cni-dir\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570379 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/061dda5b-09a3-493d-9c16-6d25055d5b82-socket-dir\") pod \"aws-ebs-csi-driver-node-hwk49\" (UID: \"061dda5b-09a3-493d-9c16-6d25055d5b82\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hwk49" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570397 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10b4d740-fb6d-486e-b6c1-5c6115a10496-host\") pod \"node-ca-kvsqc\" (UID: \"10b4d740-fb6d-486e-b6c1-5c6115a10496\") " pod="openshift-image-registry/node-ca-kvsqc" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570401 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc2355fd-e26c-4d96-b399-5528863ba211-system-cni-dir\") pod \"multus-additional-cni-plugins-cl4bq\" (UID: \"fc2355fd-e26c-4d96-b399-5528863ba211\") " pod="openshift-multus/multus-additional-cni-plugins-cl4bq" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570424 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7af4112-7294-4d61-95bd-d05625f88bdc-host-slash\") pod \"iptables-alerter-4rj8r\" (UID: \"e7af4112-7294-4d61-95bd-d05625f88bdc\") " pod="openshift-network-operator/iptables-alerter-4rj8r" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570432 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-sysctl-d\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570447 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-slash\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570452 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10b4d740-fb6d-486e-b6c1-5c6115a10496-host\") pod \"node-ca-kvsqc\" (UID: \"10b4d740-fb6d-486e-b6c1-5c6115a10496\") " pod="openshift-image-registry/node-ca-kvsqc" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570470 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-lib-modules\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570475 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7af4112-7294-4d61-95bd-d05625f88bdc-host-slash\") pod \"iptables-alerter-4rj8r\" (UID: \"e7af4112-7294-4d61-95bd-d05625f88bdc\") " pod="openshift-network-operator/iptables-alerter-4rj8r" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570472 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-system-cni-dir\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570495 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-slash\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570512 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-run-netns\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570534 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-run-netns\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570513 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-system-cni-dir\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570533 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570590 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-cni-netd\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570606 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570646 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-cni-netd\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570648 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570653 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k2v62\" (UID: \"dcc2a3e4-911a-4d26-bc60-65fc4b77cbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570667 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-socket-dir-parent\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570683 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-etc-kubernetes\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570699 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-sysconfig\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570740 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-multus-socket-dir-parent\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570753 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9aa4593-872a-4501-9a7e-8cdeec5a92bf-etc-kubernetes\") pod \"multus-8l62c\" (UID: \"f9aa4593-872a-4501-9a7e-8cdeec5a92bf\") " pod="openshift-multus/multus-8l62c" Oct 30 12:51:47.570752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.570760 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f35c9e60-a8e9-43b2-bea6-614f6555bc03-etc-sysconfig\") pod \"tuned-z9f5d\" (UID: \"f35c9e60-a8e9-43b2-bea6-614f6555bc03\") " pod="openshift-cluster-node-tuning-operator/tuned-z9f5d" Oct 30 12:51:47.572630 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.572337 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b553c14-1755-44ed-8dec-a2d1d8259c1d-metrics-certs\") pod \"network-metrics-daemon-7l5g4\" (UID: \"6b553c14-1755-44ed-8dec-a2d1d8259c1d\") " pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:47.641071 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.641037 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l5g4" Oct 30 12:51:47.650048 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.650028 4503 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:51:47.960844 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:47.960807 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7l5g4"] Oct 30 12:51:47.963697 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:47.963665 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b553c14_1755_44ed_8dec_a2d1d8259c1d.slice/crio-a2be96b648a127744038d436c1b0820bd17c130f3a0b0d1f92b509cc7742ae24 WatchSource:0}: Error finding container a2be96b648a127744038d436c1b0820bd17c130f3a0b0d1f92b509cc7742ae24: Status 404 returned error can't find the container with id a2be96b648a127744038d436c1b0820bd17c130f3a0b0d1f92b509cc7742ae24 Oct 30 12:51:48.075209 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:48.075172 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:48.075356 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:51:48.075314 4503 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:51:48.075413 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:51:48.075386 4503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls podName:7aad1c89-4805-4c51-a9dc-26db67c46f46 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:49.07536749 +0000 UTC m=+2.898595769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbsfb" (UID: "7aad1c89-4805-4c51-a9dc-26db67c46f46") : secret "insights-runtime-extractor-tls" not found Oct 30 12:51:48.404198 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:48.404165 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7l5g4" event={"ID":"6b553c14-1755-44ed-8dec-a2d1d8259c1d","Type":"ContainerStarted","Data":"a2be96b648a127744038d436c1b0820bd17c130f3a0b0d1f92b509cc7742ae24"} Oct 30 12:51:48.407034 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:48.406932 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b7kx7" event={"ID":"6e7c1377-fb2f-4b5f-83a7-b799206fb344","Type":"ContainerStarted","Data":"ae92593bf5259d2937dd307dcdbc370b473b2a5f9d67cd3f6fab04fdaeaac030"} Oct 30 12:51:49.081369 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:49.081321 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:49.081509 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:51:49.081455 4503 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 12:51:49.081543 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:51:49.081525 4503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls podName:7aad1c89-4805-4c51-a9dc-26db67c46f46 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:51.081510733 +0000 UTC m=+4.904738988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hbsfb" (UID: "7aad1c89-4805-4c51-a9dc-26db67c46f46") : secret "insights-runtime-extractor-tls" not found Oct 30 12:51:49.412264 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:49.412228 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7l5g4" event={"ID":"6b553c14-1755-44ed-8dec-a2d1d8259c1d","Type":"ContainerStarted","Data":"360dd6d11b3f41a45a48c21ed063e52904aea74f5fbd46ea54b550e072bfae7c"} Oct 30 12:51:49.414300 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:49.414277 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b7kx7" event={"ID":"6e7c1377-fb2f-4b5f-83a7-b799206fb344","Type":"ContainerStarted","Data":"6401fe2b283744fc065773dd967844b31652562ad65e70c946bec15d37000467"} Oct 30 12:51:49.414414 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:49.414397 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:50.419194 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:50.419155 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7l5g4" event={"ID":"6b553c14-1755-44ed-8dec-a2d1d8259c1d","Type":"ContainerStarted","Data":"40b53be9b5e86cbcc944b31d12ed18e1117c708fa9eeb31a104b9e772e124490"} Oct 30 12:51:51.095045 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:51.095008 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:51.097457 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:51.097431 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7aad1c89-4805-4c51-a9dc-26db67c46f46-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hbsfb\" (UID: \"7aad1c89-4805-4c51-a9dc-26db67c46f46\") " pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:51.243056 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:51.243030 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:51.248761 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:51.248738 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hbsfb" Oct 30 12:51:51.542223 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:51.542199 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hbsfb"] Oct 30 12:51:51.543656 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:51.543631 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aad1c89_4805_4c51_a9dc_26db67c46f46.slice/crio-d056d02778ba57363a7aa7366c9428fca422b7fd7156f170a69db162f59195f6 WatchSource:0}: Error finding container d056d02778ba57363a7aa7366c9428fca422b7fd7156f170a69db162f59195f6: Status 404 returned error can't find the container with id d056d02778ba57363a7aa7366c9428fca422b7fd7156f170a69db162f59195f6 Oct 30 12:51:52.427600 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:52.427548 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hbsfb" event={"ID":"7aad1c89-4805-4c51-a9dc-26db67c46f46","Type":"ContainerStarted","Data":"71c49ef88a029b903b8f5ffdfb9f37735c2f40652474d596e7b1c19f09b9b76b"} Oct 30 12:51:52.427600 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:52.427604 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hbsfb" event={"ID":"7aad1c89-4805-4c51-a9dc-26db67c46f46","Type":"ContainerStarted","Data":"42d021869881d03ad0111578714493b3de2aa27680ed200bb86be19ad1260087"} Oct 30 12:51:52.427814 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:52.427624 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hbsfb" event={"ID":"7aad1c89-4805-4c51-a9dc-26db67c46f46","Type":"ContainerStarted","Data":"d056d02778ba57363a7aa7366c9428fca422b7fd7156f170a69db162f59195f6"} Oct 30 12:51:53.432272 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:53.432244 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hbsfb" event={"ID":"7aad1c89-4805-4c51-a9dc-26db67c46f46","Type":"ContainerStarted","Data":"62383ab0f3a63e4cc5c8939f94c9abb2c2b65c61837d2612cfc09c492495de87"} Oct 30 12:51:54.130436 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:54.130380 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:51:56.240392 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.240358 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7jdw7"] Oct 30 12:51:56.242440 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.242416 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7jdw7" Oct 30 12:51:56.252438 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.252413 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-g7dlx"] Oct 30 12:51:56.254393 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.254372 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-g7dlx" Oct 30 12:51:56.264499 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.264469 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 12:51:56.264636 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.264553 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-n9cmm\"" Oct 30 12:51:56.270208 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.270182 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7jdw7"] Oct 30 12:51:56.276392 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.276366 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-g7dlx"] Oct 30 12:51:56.276713 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.276695 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 12:51:56.276831 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.276721 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 12:51:56.334175 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.334145 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e69c2260-dd1b-4a31-b08b-0e6681fa76c1-tmp\") pod \"downloads-6dd789c8b7-g7dlx\" (UID: \"e69c2260-dd1b-4a31-b08b-0e6681fa76c1\") " pod="openshift-console/downloads-6dd789c8b7-g7dlx" Oct 30 12:51:56.334344 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.334185 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlzwp\" (UniqueName: \"kubernetes.io/projected/e69c2260-dd1b-4a31-b08b-0e6681fa76c1-kube-api-access-vlzwp\") pod \"downloads-6dd789c8b7-g7dlx\" (UID: \"e69c2260-dd1b-4a31-b08b-0e6681fa76c1\") " pod="openshift-console/downloads-6dd789c8b7-g7dlx" Oct 30 12:51:56.334344 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.334219 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3df4074f-8df3-47a4-8aa8-7618640bc487-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7jdw7\" (UID: \"3df4074f-8df3-47a4-8aa8-7618640bc487\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7jdw7" Oct 30 12:51:56.334741 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.334720 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57c578b6f7-d78ps"] Oct 30 12:51:56.336422 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.336408 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.366346 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.366303 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-568p2\"" Oct 30 12:51:56.366550 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.366323 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 12:51:56.366612 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.366329 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 12:51:56.366612 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.366355 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 12:51:56.366612 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.366401 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cglsl\"" Oct 30 12:51:56.366612 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.366484 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 12:51:56.367036 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.367015 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57c578b6f7-d78ps"] Oct 30 12:51:56.434793 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.434765 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-ca-trust-extracted\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.434990 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.434806 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e69c2260-dd1b-4a31-b08b-0e6681fa76c1-tmp\") pod \"downloads-6dd789c8b7-g7dlx\" (UID: \"e69c2260-dd1b-4a31-b08b-0e6681fa76c1\") " pod="openshift-console/downloads-6dd789c8b7-g7dlx" Oct 30 12:51:56.434990 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.434822 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlzwp\" (UniqueName: \"kubernetes.io/projected/e69c2260-dd1b-4a31-b08b-0e6681fa76c1-kube-api-access-vlzwp\") pod \"downloads-6dd789c8b7-g7dlx\" (UID: \"e69c2260-dd1b-4a31-b08b-0e6681fa76c1\") " pod="openshift-console/downloads-6dd789c8b7-g7dlx" Oct 30 12:51:56.434990 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.434840 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-image-registry-private-configuration\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.434990 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.434887 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-registry-certificates\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.434990 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.434933 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-trusted-ca\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.435157 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.434994 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-installation-pull-secrets\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.435157 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.435063 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3df4074f-8df3-47a4-8aa8-7618640bc487-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7jdw7\" (UID: \"3df4074f-8df3-47a4-8aa8-7618640bc487\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7jdw7" Oct 30 12:51:56.435157 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.435137 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-bound-sa-token\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.435246 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.435199 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-registry-tls\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.435246 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.435224 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfhjw\" (UniqueName: \"kubernetes.io/projected/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-kube-api-access-qfhjw\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.435338 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.435320 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e69c2260-dd1b-4a31-b08b-0e6681fa76c1-tmp\") pod \"downloads-6dd789c8b7-g7dlx\" (UID: \"e69c2260-dd1b-4a31-b08b-0e6681fa76c1\") " pod="openshift-console/downloads-6dd789c8b7-g7dlx" Oct 30 12:51:56.437487 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.437465 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3df4074f-8df3-47a4-8aa8-7618640bc487-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7jdw7\" (UID: \"3df4074f-8df3-47a4-8aa8-7618640bc487\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7jdw7" Oct 30 12:51:56.459621 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.459587 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlzwp\" (UniqueName: \"kubernetes.io/projected/e69c2260-dd1b-4a31-b08b-0e6681fa76c1-kube-api-access-vlzwp\") pod \"downloads-6dd789c8b7-g7dlx\" (UID: \"e69c2260-dd1b-4a31-b08b-0e6681fa76c1\") " pod="openshift-console/downloads-6dd789c8b7-g7dlx" Oct 30 12:51:56.536241 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.536149 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-ca-trust-extracted\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.536241 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.536194 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-image-registry-private-configuration\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.536241 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.536214 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-registry-certificates\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.536505 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.536424 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-trusted-ca\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.536505 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.536466 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-installation-pull-secrets\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.536505 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.536501 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-bound-sa-token\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.536746 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.536529 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-registry-tls\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.536746 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.536553 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfhjw\" (UniqueName: \"kubernetes.io/projected/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-kube-api-access-qfhjw\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.536878 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.536840 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-ca-trust-extracted\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.537234 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.537208 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-registry-certificates\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.537624 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.537601 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-trusted-ca\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.538847 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.538828 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-image-registry-private-configuration\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.539678 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.539655 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-installation-pull-secrets\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.539752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.539686 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-registry-tls\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.554123 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.554101 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7jdw7" Oct 30 12:51:56.563730 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.563706 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-g7dlx" Oct 30 12:51:56.566041 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.565889 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-bound-sa-token\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.567894 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.567872 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfhjw\" (UniqueName: \"kubernetes.io/projected/3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe-kube-api-access-qfhjw\") pod \"image-registry-57c578b6f7-d78ps\" (UID: \"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe\") " pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.646290 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.646260 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:56.863306 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.863281 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7jdw7"] Oct 30 12:51:56.865093 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:56.865067 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df4074f_8df3_47a4_8aa8_7618640bc487.slice/crio-7ffa2bec9748e6a34e0b10c9abd1cdcd8e88f1bb6bcdecb40286e153997d09d4 WatchSource:0}: Error finding container 7ffa2bec9748e6a34e0b10c9abd1cdcd8e88f1bb6bcdecb40286e153997d09d4: Status 404 returned error can't find the container with id 7ffa2bec9748e6a34e0b10c9abd1cdcd8e88f1bb6bcdecb40286e153997d09d4 Oct 30 12:51:56.871182 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.871160 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-g7dlx"] Oct 30 12:51:56.876925 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:56.876900 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69c2260_dd1b_4a31_b08b_0e6681fa76c1.slice/crio-8e980576ad86317ee600e83a02b4731a3d324862ac2cffc626d6bd863fe4f13c WatchSource:0}: Error finding container 8e980576ad86317ee600e83a02b4731a3d324862ac2cffc626d6bd863fe4f13c: Status 404 returned error can't find the container with id 8e980576ad86317ee600e83a02b4731a3d324862ac2cffc626d6bd863fe4f13c Oct 30 12:51:56.948001 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:56.947977 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57c578b6f7-d78ps"] Oct 30 12:51:56.949406 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:51:56.949380 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8ee171_fa17_418f_a59b_2cc6bd5aa2fe.slice/crio-b62e6b456be8284dc761a7c1a2db6a6eff06c87d978837c4168ed6fa2f82d313 WatchSource:0}: Error finding container b62e6b456be8284dc761a7c1a2db6a6eff06c87d978837c4168ed6fa2f82d313: Status 404 returned error can't find the container with id b62e6b456be8284dc761a7c1a2db6a6eff06c87d978837c4168ed6fa2f82d313 Oct 30 12:51:57.447370 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:57.447333 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" event={"ID":"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe","Type":"ContainerStarted","Data":"772e6c26eeb6d2504bf756bbc0dcc021d4ad8eeaf4532045b526e415aef6528d"} Oct 30 12:51:57.447830 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:57.447387 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" event={"ID":"3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe","Type":"ContainerStarted","Data":"b62e6b456be8284dc761a7c1a2db6a6eff06c87d978837c4168ed6fa2f82d313"} Oct 30 12:51:57.447830 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:57.447426 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:51:57.449062 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:57.449033 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-g7dlx" event={"ID":"e69c2260-dd1b-4a31-b08b-0e6681fa76c1","Type":"ContainerStarted","Data":"8e980576ad86317ee600e83a02b4731a3d324862ac2cffc626d6bd863fe4f13c"} Oct 30 12:51:57.450359 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:57.450306 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7jdw7" event={"ID":"3df4074f-8df3-47a4-8aa8-7618640bc487","Type":"ContainerStarted","Data":"7ffa2bec9748e6a34e0b10c9abd1cdcd8e88f1bb6bcdecb40286e153997d09d4"} Oct 30 12:51:57.496030 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:57.495328 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" podStartSLOduration=1.49530911 podStartE2EDuration="1.49530911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:57.495041235 +0000 UTC m=+11.318269534" watchObservedRunningTime="2026-10-30 12:51:57.49530911 +0000 UTC m=+11.318537385" Oct 30 12:51:57.522522 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:57.521355 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:51:57.522522 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:57.521574 4503 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 12:51:58.455769 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:58.455725 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7jdw7" event={"ID":"3df4074f-8df3-47a4-8aa8-7618640bc487","Type":"ContainerStarted","Data":"f654d51eb94b58ddd61e83a902d23e0ba64fd9065b54f6154223c693dbb4fb24"} Oct 30 12:51:58.497213 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:58.497162 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7jdw7" podStartSLOduration=1.621758667 podStartE2EDuration="2.497149029s" totalImagesPullingTime="875.390362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:58.496598373 +0000 UTC m=+12.319826649" watchObservedRunningTime="2026-10-30 12:51:58.497149029 +0000 UTC m=+12.320377298" Oct 30 12:51:59.421478 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:59.421443 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b7kx7" Oct 30 12:51:59.459622 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:59.459586 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7jdw7" Oct 30 12:51:59.465081 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:51:59.465053 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7jdw7" Oct 30 12:51:59.705160 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:51:59.705076 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 12:52:02.869284 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:02.869252 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2v62" Oct 30 12:52:04.697805 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.697768 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd"] Oct 30 12:52:04.700978 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.700479 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd" Oct 30 12:52:04.721349 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.721307 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-285cb\"" Oct 30 12:52:04.721470 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.721406 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 12:52:04.721527 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.721486 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 12:52:04.733849 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.733824 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 12:52:04.734045 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.734023 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 12:52:04.734202 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.734157 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 12:52:04.735058 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.735038 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd"] Oct 30 12:52:04.747264 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.747240 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dlsxw"] Oct 30 12:52:04.749704 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.749685 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:04.768184 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.768160 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 12:52:04.768315 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.768284 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 12:52:04.768376 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.768355 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rwc4t\"" Oct 30 12:52:04.799205 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.799164 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cd66bfb-6bb5-46e1-b0ae-91f0520222ed-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6nlzd\" (UID: \"3cd66bfb-6bb5-46e1-b0ae-91f0520222ed\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd" Oct 30 12:52:04.799370 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.799217 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cphqt\" (UniqueName: \"kubernetes.io/projected/3cd66bfb-6bb5-46e1-b0ae-91f0520222ed-kube-api-access-cphqt\") pod \"openshift-state-metrics-7bd7489957-6nlzd\" (UID: \"3cd66bfb-6bb5-46e1-b0ae-91f0520222ed\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd" Oct 30 12:52:04.799370 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.799247 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cd66bfb-6bb5-46e1-b0ae-91f0520222ed-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6nlzd\" (UID: \"3cd66bfb-6bb5-46e1-b0ae-91f0520222ed\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd" Oct 30 12:52:04.799370 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.799268 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cd66bfb-6bb5-46e1-b0ae-91f0520222ed-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6nlzd\" (UID: \"3cd66bfb-6bb5-46e1-b0ae-91f0520222ed\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd" Oct 30 12:52:04.823314 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.823264 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 12:52:04.900614 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.900578 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cd66bfb-6bb5-46e1-b0ae-91f0520222ed-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6nlzd\" (UID: \"3cd66bfb-6bb5-46e1-b0ae-91f0520222ed\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd" Oct 30 12:52:04.900801 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.900628 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/933dffcb-401a-42ad-8233-d1cf1ce231e3-node-exporter-accelerators-collector-config\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:04.900801 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.900654 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cd66bfb-6bb5-46e1-b0ae-91f0520222ed-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6nlzd\" (UID: \"3cd66bfb-6bb5-46e1-b0ae-91f0520222ed\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd" Oct 30 12:52:04.900801 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.900674 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/933dffcb-401a-42ad-8233-d1cf1ce231e3-root\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:04.900801 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.900748 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/933dffcb-401a-42ad-8233-d1cf1ce231e3-node-exporter-textfile\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:04.900801 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.900786 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/933dffcb-401a-42ad-8233-d1cf1ce231e3-node-exporter-wtmp\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:04.901107 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.900887 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/933dffcb-401a-42ad-8233-d1cf1ce231e3-node-exporter-tls\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:04.901107 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.900928 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/933dffcb-401a-42ad-8233-d1cf1ce231e3-metrics-client-ca\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:04.901107 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.900980 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62wbk\" (UniqueName: \"kubernetes.io/projected/933dffcb-401a-42ad-8233-d1cf1ce231e3-kube-api-access-62wbk\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:04.901107 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.901009 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cd66bfb-6bb5-46e1-b0ae-91f0520222ed-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6nlzd\" (UID: \"3cd66bfb-6bb5-46e1-b0ae-91f0520222ed\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd" Oct 30 12:52:04.901107 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.901050 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/933dffcb-401a-42ad-8233-d1cf1ce231e3-sys\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:04.901107 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.901088 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/933dffcb-401a-42ad-8233-d1cf1ce231e3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:04.901421 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.901146 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cphqt\" (UniqueName: \"kubernetes.io/projected/3cd66bfb-6bb5-46e1-b0ae-91f0520222ed-kube-api-access-cphqt\") pod \"openshift-state-metrics-7bd7489957-6nlzd\" (UID: \"3cd66bfb-6bb5-46e1-b0ae-91f0520222ed\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd" Oct 30 12:52:04.901505 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.901477 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cd66bfb-6bb5-46e1-b0ae-91f0520222ed-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6nlzd\" (UID: \"3cd66bfb-6bb5-46e1-b0ae-91f0520222ed\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd" Oct 30 12:52:04.903589 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.903562 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cd66bfb-6bb5-46e1-b0ae-91f0520222ed-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6nlzd\" (UID: \"3cd66bfb-6bb5-46e1-b0ae-91f0520222ed\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd" Oct 30 12:52:04.903982 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.903964 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cd66bfb-6bb5-46e1-b0ae-91f0520222ed-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6nlzd\" (UID: \"3cd66bfb-6bb5-46e1-b0ae-91f0520222ed\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd" Oct 30 12:52:04.927946 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:04.927912 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cphqt\" (UniqueName: \"kubernetes.io/projected/3cd66bfb-6bb5-46e1-b0ae-91f0520222ed-kube-api-access-cphqt\") pod \"openshift-state-metrics-7bd7489957-6nlzd\" (UID: \"3cd66bfb-6bb5-46e1-b0ae-91f0520222ed\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd" Oct 30 12:52:05.002023 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.001925 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/933dffcb-401a-42ad-8233-d1cf1ce231e3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.002023 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.002013 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/933dffcb-401a-42ad-8233-d1cf1ce231e3-node-exporter-accelerators-collector-config\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.003342 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.002504 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/933dffcb-401a-42ad-8233-d1cf1ce231e3-root\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.003342 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.002535 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/933dffcb-401a-42ad-8233-d1cf1ce231e3-node-exporter-accelerators-collector-config\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.003342 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.002553 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/933dffcb-401a-42ad-8233-d1cf1ce231e3-node-exporter-textfile\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.003342 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.002594 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/933dffcb-401a-42ad-8233-d1cf1ce231e3-node-exporter-wtmp\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.003342 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.002607 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/933dffcb-401a-42ad-8233-d1cf1ce231e3-root\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.003342 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.002626 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/933dffcb-401a-42ad-8233-d1cf1ce231e3-node-exporter-tls\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.003342 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.002658 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/933dffcb-401a-42ad-8233-d1cf1ce231e3-metrics-client-ca\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.003342 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.002684 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62wbk\" (UniqueName: \"kubernetes.io/projected/933dffcb-401a-42ad-8233-d1cf1ce231e3-kube-api-access-62wbk\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.003342 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.002741 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/933dffcb-401a-42ad-8233-d1cf1ce231e3-sys\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.003342 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.002798 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/933dffcb-401a-42ad-8233-d1cf1ce231e3-node-exporter-textfile\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.003342 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.002821 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/933dffcb-401a-42ad-8233-d1cf1ce231e3-sys\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.003342 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:52:05.002880 4503 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Oct 30 12:52:05.003342 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.003290 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/933dffcb-401a-42ad-8233-d1cf1ce231e3-metrics-client-ca\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.003342 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.003328 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/933dffcb-401a-42ad-8233-d1cf1ce231e3-node-exporter-wtmp\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.004140 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:52:05.003375 4503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/933dffcb-401a-42ad-8233-d1cf1ce231e3-node-exporter-tls podName:933dffcb-401a-42ad-8233-d1cf1ce231e3 nodeName:}" failed. No retries permitted until 2026-10-30 12:52:05.502927426 +0000 UTC m=+19.326155696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/933dffcb-401a-42ad-8233-d1cf1ce231e3-node-exporter-tls") pod "node-exporter-dlsxw" (UID: "933dffcb-401a-42ad-8233-d1cf1ce231e3") : secret "node-exporter-tls" not found Oct 30 12:52:05.004805 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.004737 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/933dffcb-401a-42ad-8233-d1cf1ce231e3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.014858 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.014833 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd" Oct 30 12:52:05.030424 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.030401 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62wbk\" (UniqueName: \"kubernetes.io/projected/933dffcb-401a-42ad-8233-d1cf1ce231e3-kube-api-access-62wbk\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.336153 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.336119 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd"] Oct 30 12:52:05.338291 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:52:05.338256 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd66bfb_6bb5_46e1_b0ae_91f0520222ed.slice/crio-d101299b6be165c436085ff299ab0845622c7403932d6c4e963855379606a25c WatchSource:0}: Error finding container d101299b6be165c436085ff299ab0845622c7403932d6c4e963855379606a25c: Status 404 returned error can't find the container with id d101299b6be165c436085ff299ab0845622c7403932d6c4e963855379606a25c Oct 30 12:52:05.482958 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.482912 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd" event={"ID":"3cd66bfb-6bb5-46e1-b0ae-91f0520222ed","Type":"ContainerStarted","Data":"298be14447452ec5a60ae3c975a9bb56a28d4e691bc4f8fb6e9642fbbfc01471"} Oct 30 12:52:05.483102 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.483038 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd" event={"ID":"3cd66bfb-6bb5-46e1-b0ae-91f0520222ed","Type":"ContainerStarted","Data":"f0ee9f9bf48e137262a2315cf46f89ee4ed941098434de574ab321a59f769c34"} Oct 30 12:52:05.483102 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.483070 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd" event={"ID":"3cd66bfb-6bb5-46e1-b0ae-91f0520222ed","Type":"ContainerStarted","Data":"d101299b6be165c436085ff299ab0845622c7403932d6c4e963855379606a25c"} Oct 30 12:52:05.507298 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.507265 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/933dffcb-401a-42ad-8233-d1cf1ce231e3-node-exporter-tls\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.510484 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.510458 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/933dffcb-401a-42ad-8233-d1cf1ce231e3-node-exporter-tls\") pod \"node-exporter-dlsxw\" (UID: \"933dffcb-401a-42ad-8233-d1cf1ce231e3\") " pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.663627 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:05.663586 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dlsxw" Oct 30 12:52:05.673768 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:52:05.673740 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933dffcb_401a_42ad_8233_d1cf1ce231e3.slice/crio-1716119b608ce48c411b6a63a7c397113a7d6576523a712db576b88221dfa127 WatchSource:0}: Error finding container 1716119b608ce48c411b6a63a7c397113a7d6576523a712db576b88221dfa127: Status 404 returned error can't find the container with id 1716119b608ce48c411b6a63a7c397113a7d6576523a712db576b88221dfa127 Oct 30 12:52:06.496161 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:06.496120 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd" event={"ID":"3cd66bfb-6bb5-46e1-b0ae-91f0520222ed","Type":"ContainerStarted","Data":"d3d815372592f3b69a12547fa53dad51580116db04cb3ef4e5ae49aca71de856"} Oct 30 12:52:06.498357 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:06.498326 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dlsxw" event={"ID":"933dffcb-401a-42ad-8233-d1cf1ce231e3","Type":"ContainerStarted","Data":"1716119b608ce48c411b6a63a7c397113a7d6576523a712db576b88221dfa127"} Oct 30 12:52:06.552805 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:06.552741 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6nlzd" podStartSLOduration=1.655783556 podStartE2EDuration="2.552722438s" totalImagesPullingTime="896.938882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:06.55198899 +0000 UTC m=+20.375217267" watchObservedRunningTime="2026-10-30 12:52:06.552722438 +0000 UTC m=+20.375950711" Oct 30 12:52:06.895580 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:06.895548 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-696fdb48d6-54hj7"] Oct 30 12:52:06.898475 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:06.898453 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:06.926348 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:06.926309 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-696fdb48d6-54hj7"] Oct 30 12:52:06.986491 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:06.986454 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 12:52:06.986491 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:06.986480 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 12:52:06.986718 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:06.986593 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 12:52:06.986718 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:06.986628 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 12:52:06.986718 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:06.986636 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7k4tncb7564eo\"" Oct 30 12:52:06.986718 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:06.986665 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 12:52:06.986842 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:06.986729 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ljsk7\"" Oct 30 12:52:07.021028 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.020994 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9bd07a93-063c-4c0e-bff6-afd530af1fe2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.021199 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.021044 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9bd07a93-063c-4c0e-bff6-afd530af1fe2-secret-thanos-querier-tls\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.021199 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.021161 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9bd07a93-063c-4c0e-bff6-afd530af1fe2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.021302 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.021202 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9bd07a93-063c-4c0e-bff6-afd530af1fe2-secret-grpc-tls\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.021302 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.021234 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9bd07a93-063c-4c0e-bff6-afd530af1fe2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.021302 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.021272 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bd07a93-063c-4c0e-bff6-afd530af1fe2-metrics-client-ca\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.021417 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.021297 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp9vb\" (UniqueName: \"kubernetes.io/projected/9bd07a93-063c-4c0e-bff6-afd530af1fe2-kube-api-access-kp9vb\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.021417 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.021344 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9bd07a93-063c-4c0e-bff6-afd530af1fe2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.021417 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.021381 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bd07a93-063c-4c0e-bff6-afd530af1fe2-tmp\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.122448 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.122406 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bd07a93-063c-4c0e-bff6-afd530af1fe2-tmp\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.122639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.122481 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9bd07a93-063c-4c0e-bff6-afd530af1fe2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.122639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.122513 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9bd07a93-063c-4c0e-bff6-afd530af1fe2-secret-thanos-querier-tls\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.122639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.122553 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9bd07a93-063c-4c0e-bff6-afd530af1fe2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.122639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.122576 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9bd07a93-063c-4c0e-bff6-afd530af1fe2-secret-grpc-tls\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.122639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.122600 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9bd07a93-063c-4c0e-bff6-afd530af1fe2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.122639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.122635 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bd07a93-063c-4c0e-bff6-afd530af1fe2-metrics-client-ca\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.122898 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.122660 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9vb\" (UniqueName: \"kubernetes.io/projected/9bd07a93-063c-4c0e-bff6-afd530af1fe2-kube-api-access-kp9vb\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.122898 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.122704 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9bd07a93-063c-4c0e-bff6-afd530af1fe2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.122898 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.122839 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bd07a93-063c-4c0e-bff6-afd530af1fe2-tmp\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.123804 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.123765 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bd07a93-063c-4c0e-bff6-afd530af1fe2-metrics-client-ca\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.125822 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.125797 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9bd07a93-063c-4c0e-bff6-afd530af1fe2-secret-thanos-querier-tls\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.125925 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.125815 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9bd07a93-063c-4c0e-bff6-afd530af1fe2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.126029 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.126003 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9bd07a93-063c-4c0e-bff6-afd530af1fe2-secret-grpc-tls\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.126427 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.126391 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9bd07a93-063c-4c0e-bff6-afd530af1fe2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.126956 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.126919 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9bd07a93-063c-4c0e-bff6-afd530af1fe2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.127027 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.126981 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9bd07a93-063c-4c0e-bff6-afd530af1fe2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.149170 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.149109 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp9vb\" (UniqueName: \"kubernetes.io/projected/9bd07a93-063c-4c0e-bff6-afd530af1fe2-kube-api-access-kp9vb\") pod \"thanos-querier-696fdb48d6-54hj7\" (UID: \"9bd07a93-063c-4c0e-bff6-afd530af1fe2\") " pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.218967 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.218915 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:07.503901 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.503822 4503 generic.go:356] "Generic (PLEG): container finished" podID="933dffcb-401a-42ad-8233-d1cf1ce231e3" containerID="7aa9b306dded0cf16cc6e2b05422ddaa8a9a11063a7cf49ec503898ecaa4fa8e" exitCode=0 Oct 30 12:52:07.503901 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.503865 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dlsxw" event={"ID":"933dffcb-401a-42ad-8233-d1cf1ce231e3","Type":"ContainerDied","Data":"7aa9b306dded0cf16cc6e2b05422ddaa8a9a11063a7cf49ec503898ecaa4fa8e"} Oct 30 12:52:07.550585 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:07.550048 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-696fdb48d6-54hj7"] Oct 30 12:52:07.554121 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:52:07.554094 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd07a93_063c_4c0e_bff6_afd530af1fe2.slice/crio-82505ec0cb073fe5530e7bfeeb389347d3767ce352773fab93e347110491c773 WatchSource:0}: Error finding container 82505ec0cb073fe5530e7bfeeb389347d3767ce352773fab93e347110491c773: Status 404 returned error can't find the container with id 82505ec0cb073fe5530e7bfeeb389347d3767ce352773fab93e347110491c773 Oct 30 12:52:08.510886 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:08.510847 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dlsxw" event={"ID":"933dffcb-401a-42ad-8233-d1cf1ce231e3","Type":"ContainerStarted","Data":"983dafb2441e1673940d0d092fcc6c7e3cfc28b60a463e43f353f8440a9f27b9"} Oct 30 12:52:08.511358 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:08.510909 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dlsxw" event={"ID":"933dffcb-401a-42ad-8233-d1cf1ce231e3","Type":"ContainerStarted","Data":"df09ca85d3ccd65b1c1afde28aaf8050ff7ae2f9efbc01db22f406ca60ce5ce9"} Oct 30 12:52:08.513189 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:08.513160 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" event={"ID":"9bd07a93-063c-4c0e-bff6-afd530af1fe2","Type":"ContainerStarted","Data":"82505ec0cb073fe5530e7bfeeb389347d3767ce352773fab93e347110491c773"} Oct 30 12:52:08.585993 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:08.585912 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dlsxw" podStartSLOduration=3.611029145 podStartE2EDuration="4.585893607s" totalImagesPullingTime="896.591257ms" totalInitContainerRuntime="78.273205ms" isStatefulPod=true podCreationTimestamp="2026-10-30 12:52:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:08.573143577 +0000 UTC m=+22.396371854" watchObservedRunningTime="2026-10-30 12:52:08.585893607 +0000 UTC m=+22.409121885" Oct 30 12:52:09.520378 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:09.520348 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" event={"ID":"9bd07a93-063c-4c0e-bff6-afd530af1fe2","Type":"ContainerStarted","Data":"5eca21abb68aae8c9931c6ffd92d0160625ddc8b7da8fa5650f27eccb6838c09"} Oct 30 12:52:10.526699 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:10.526611 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" event={"ID":"9bd07a93-063c-4c0e-bff6-afd530af1fe2","Type":"ContainerStarted","Data":"a5ffe9ec74261a5221fb93b6acc6a9d7b6192f9dc4f2a8e2f0f8ea1ab0a870e6"} Oct 30 12:52:10.527150 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:10.526716 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" event={"ID":"9bd07a93-063c-4c0e-bff6-afd530af1fe2","Type":"ContainerStarted","Data":"4b96e8b6bc186b08ee638f75fcd5c98757bde9e1618aff4356387ea0690d234f"} Oct 30 12:52:10.527150 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:10.526759 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" event={"ID":"9bd07a93-063c-4c0e-bff6-afd530af1fe2","Type":"ContainerStarted","Data":"45dd081f1d7325c1a322a2e41bfbe014a0884c03d6657da6c054c118066ed19b"} Oct 30 12:52:10.527150 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:10.526798 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" event={"ID":"9bd07a93-063c-4c0e-bff6-afd530af1fe2","Type":"ContainerStarted","Data":"8eebc138cb3a6f9d36dc9405bb4e42a9d9ecc608d4b5aefdbb8d665f9be9e549"} Oct 30 12:52:10.527150 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:10.526845 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:10.527150 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:10.526893 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" event={"ID":"9bd07a93-063c-4c0e-bff6-afd530af1fe2","Type":"ContainerStarted","Data":"e93c6d74be1acdc653b1932e550926ae16c622ee462eb387a76e11d01be9e930"} Oct 30 12:52:10.603094 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:10.603040 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" podStartSLOduration=2.086560613 podStartE2EDuration="4.603022306s" totalImagesPullingTime="2.516461693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:10.58996951 +0000 UTC m=+24.413197788" watchObservedRunningTime="2026-10-30 12:52:10.603022306 +0000 UTC m=+24.426250585" Oct 30 12:52:13.001874 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:13.001837 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pz66k" Oct 30 12:52:13.540678 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:13.540632 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-g7dlx" event={"ID":"e69c2260-dd1b-4a31-b08b-0e6681fa76c1","Type":"ContainerStarted","Data":"37ea860498ed7342b9ef2a9968d2116d158a58d9fc02015fa1f055580290df50"} Oct 30 12:52:13.540902 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:13.540886 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-g7dlx" Oct 30 12:52:13.553304 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:13.553272 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-g7dlx" Oct 30 12:52:13.616349 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:13.616260 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-g7dlx" podStartSLOduration=1.462301184 podStartE2EDuration="17.616236733s" totalImagesPullingTime="16.153935549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:13.602995311 +0000 UTC m=+27.426223590" watchObservedRunningTime="2026-10-30 12:52:13.616236733 +0000 UTC m=+27.439465007" Oct 30 12:52:14.134054 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:14.134019 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xb4w4" Oct 30 12:52:16.538837 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:16.538806 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-696fdb48d6-54hj7" Oct 30 12:52:18.461382 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:18.461348 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57c578b6f7-d78ps" Oct 30 12:52:29.699409 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:52:29.699324 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 12:52:44.197417 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.197383 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:52:44.219591 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.219556 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.222284 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.222257 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:52:44.237506 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.237478 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 12:52:44.237672 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.237478 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 12:52:44.237743 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.237498 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 12:52:44.237743 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.237556 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 12:52:44.237743 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.237561 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 12:52:44.237743 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.237585 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 12:52:44.237743 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.237586 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 12:52:44.237743 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.237617 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-khs4n\"" Oct 30 12:52:44.237743 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.237637 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 12:52:44.263052 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.263020 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 12:52:44.303737 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.303705 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.303737 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.303743 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15352ec1-7ce6-4cfe-8427-02727875d683-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.303918 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.303765 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q94p\" (UniqueName: \"kubernetes.io/projected/15352ec1-7ce6-4cfe-8427-02727875d683-kube-api-access-9q94p\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.303918 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.303852 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-config-volume\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.303918 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.303869 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15352ec1-7ce6-4cfe-8427-02727875d683-config-out\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.303918 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.303889 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15352ec1-7ce6-4cfe-8427-02727875d683-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.303918 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.303915 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.304160 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.304008 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-web-config\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.304160 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.304064 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.304160 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.304109 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.304160 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.304138 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.304275 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.304207 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15352ec1-7ce6-4cfe-8427-02727875d683-tls-assets\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.304275 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.304223 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/15352ec1-7ce6-4cfe-8427-02727875d683-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.405318 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.405276 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.405490 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.405328 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.405490 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.405346 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.405490 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.405373 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15352ec1-7ce6-4cfe-8427-02727875d683-tls-assets\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.405490 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.405390 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/15352ec1-7ce6-4cfe-8427-02727875d683-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.405490 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.405411 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.405490 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.405427 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15352ec1-7ce6-4cfe-8427-02727875d683-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.405490 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.405444 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q94p\" (UniqueName: \"kubernetes.io/projected/15352ec1-7ce6-4cfe-8427-02727875d683-kube-api-access-9q94p\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.405490 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.405470 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-config-volume\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.405490 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.405492 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15352ec1-7ce6-4cfe-8427-02727875d683-config-out\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.405994 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.405521 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15352ec1-7ce6-4cfe-8427-02727875d683-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.405994 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.405550 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.405994 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.405572 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-web-config\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.406169 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.406035 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/15352ec1-7ce6-4cfe-8427-02727875d683-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.406926 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.406893 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15352ec1-7ce6-4cfe-8427-02727875d683-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.407173 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.407143 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15352ec1-7ce6-4cfe-8427-02727875d683-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.408878 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.408851 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.409014 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.408890 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15352ec1-7ce6-4cfe-8427-02727875d683-config-out\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.409014 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.408997 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.409227 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.409200 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.409329 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.409263 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.409574 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.409556 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.409653 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.409636 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-web-config\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.410104 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.410089 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15352ec1-7ce6-4cfe-8427-02727875d683-tls-assets\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.410516 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.410501 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/15352ec1-7ce6-4cfe-8427-02727875d683-config-volume\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.430734 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.430712 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q94p\" (UniqueName: \"kubernetes.io/projected/15352ec1-7ce6-4cfe-8427-02727875d683-kube-api-access-9q94p\") pod \"alertmanager-main-0\" (UID: \"15352ec1-7ce6-4cfe-8427-02727875d683\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.626752 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.626670 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:44.987816 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:44.987780 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:52:45.000736 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:52:45.000704 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15352ec1_7ce6_4cfe_8427_02727875d683.slice/crio-f2fb6c75c0f9a4274d3428df0417ca78c76d7e4ca20abdf217bc26d87446780f WatchSource:0}: Error finding container f2fb6c75c0f9a4274d3428df0417ca78c76d7e4ca20abdf217bc26d87446780f: Status 404 returned error can't find the container with id f2fb6c75c0f9a4274d3428df0417ca78c76d7e4ca20abdf217bc26d87446780f Oct 30 12:52:45.651581 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:45.651545 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15352ec1-7ce6-4cfe-8427-02727875d683","Type":"ContainerStarted","Data":"f2fb6c75c0f9a4274d3428df0417ca78c76d7e4ca20abdf217bc26d87446780f"} Oct 30 12:52:46.656103 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:46.656063 4503 generic.go:356] "Generic (PLEG): container finished" podID="15352ec1-7ce6-4cfe-8427-02727875d683" containerID="5c159d5f2f725cf6f9c4897ddf8062d2e347249fbd470a2544de17d38c1cb88a" exitCode=0 Oct 30 12:52:46.656586 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:46.656109 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15352ec1-7ce6-4cfe-8427-02727875d683","Type":"ContainerDied","Data":"5c159d5f2f725cf6f9c4897ddf8062d2e347249fbd470a2544de17d38c1cb88a"} Oct 30 12:52:48.667340 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:48.667303 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15352ec1-7ce6-4cfe-8427-02727875d683","Type":"ContainerStarted","Data":"951d5ab8ee19e98e43d7ce3f730c93f0097b3525a3406d98050f087753f8b7f9"} Oct 30 12:52:48.667794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:48.667381 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15352ec1-7ce6-4cfe-8427-02727875d683","Type":"ContainerStarted","Data":"0d03cd7fb0a2e6cd7cb4be1e1963c86a77a21e5f64306634eee07912dcd6b887"} Oct 30 12:52:48.667794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:48.667421 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15352ec1-7ce6-4cfe-8427-02727875d683","Type":"ContainerStarted","Data":"47a12b39ec57f9a5b97cb01f10712763a04e843f6261aefb5873725ec1e5a401"} Oct 30 12:52:49.673892 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:49.673857 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15352ec1-7ce6-4cfe-8427-02727875d683","Type":"ContainerStarted","Data":"e84ee4fdd7e220fcc4f1aace9f111e6017e31ae76b96673da4fa78878f15d129"} Oct 30 12:52:49.674309 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:49.673921 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15352ec1-7ce6-4cfe-8427-02727875d683","Type":"ContainerStarted","Data":"2c3e5d02c0ced8574d0ea2a6e4124343b19a6b66685b692f64eb625855999677"} Oct 30 12:52:49.674309 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:49.673969 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15352ec1-7ce6-4cfe-8427-02727875d683","Type":"ContainerStarted","Data":"227646069db129a372bc3510ce395802f755bcac803254af8045868b34f3949a"} Oct 30 12:52:49.768880 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:52:49.768824 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.715190503 podStartE2EDuration="5.768808881s" totalImagesPullingTime="3.040381369s" totalInitContainerRuntime="13.237009ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:49.767881068 +0000 UTC m=+63.591109345" watchObservedRunningTime="2026-10-30 12:52:49.768808881 +0000 UTC m=+63.592037157" Oct 30 12:52:59.700454 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:52:59.700408 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 12:53:29.699798 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:53:29.699761 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 12:53:59.699546 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:53:59.699468 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 12:54:29.700168 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:54:29.700131 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 12:54:43.338104 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:43.338071 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l"] Oct 30 12:54:43.340372 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:43.340355 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l" Oct 30 12:54:43.372606 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:43.372575 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 12:54:43.372757 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:43.372663 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 12:54:43.372757 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:43.372732 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-qz9x8\"" Oct 30 12:54:43.373775 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:43.373757 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l"] Oct 30 12:54:43.417486 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:43.417446 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntt6f\" (UniqueName: \"kubernetes.io/projected/b6f03757-55c6-41d8-888a-4ccb9c9d9981-kube-api-access-ntt6f\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l\" (UID: \"b6f03757-55c6-41d8-888a-4ccb9c9d9981\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l" Oct 30 12:54:43.417667 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:43.417492 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6f03757-55c6-41d8-888a-4ccb9c9d9981-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l\" (UID: \"b6f03757-55c6-41d8-888a-4ccb9c9d9981\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l" Oct 30 12:54:43.417667 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:43.417599 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6f03757-55c6-41d8-888a-4ccb9c9d9981-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l\" (UID: \"b6f03757-55c6-41d8-888a-4ccb9c9d9981\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l" Oct 30 12:54:43.518960 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:43.518905 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntt6f\" (UniqueName: \"kubernetes.io/projected/b6f03757-55c6-41d8-888a-4ccb9c9d9981-kube-api-access-ntt6f\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l\" (UID: \"b6f03757-55c6-41d8-888a-4ccb9c9d9981\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l" Oct 30 12:54:43.519133 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:43.518980 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6f03757-55c6-41d8-888a-4ccb9c9d9981-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l\" (UID: \"b6f03757-55c6-41d8-888a-4ccb9c9d9981\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l" Oct 30 12:54:43.519133 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:43.519102 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6f03757-55c6-41d8-888a-4ccb9c9d9981-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l\" (UID: \"b6f03757-55c6-41d8-888a-4ccb9c9d9981\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l" Oct 30 12:54:43.519413 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:43.519396 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6f03757-55c6-41d8-888a-4ccb9c9d9981-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l\" (UID: \"b6f03757-55c6-41d8-888a-4ccb9c9d9981\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l" Oct 30 12:54:43.519460 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:43.519426 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6f03757-55c6-41d8-888a-4ccb9c9d9981-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l\" (UID: \"b6f03757-55c6-41d8-888a-4ccb9c9d9981\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l" Oct 30 12:54:43.544657 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:43.544628 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntt6f\" (UniqueName: \"kubernetes.io/projected/b6f03757-55c6-41d8-888a-4ccb9c9d9981-kube-api-access-ntt6f\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l\" (UID: \"b6f03757-55c6-41d8-888a-4ccb9c9d9981\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l" Oct 30 12:54:43.652782 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:43.652747 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l" Oct 30 12:54:43.972522 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:43.972491 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l"] Oct 30 12:54:43.974249 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:54:43.974208 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f03757_55c6_41d8_888a_4ccb9c9d9981.slice/crio-63c1fd9259d48d8294fb91f25d10a96b5484d2593ac8a45599db6709e5dc4b87 WatchSource:0}: Error finding container 63c1fd9259d48d8294fb91f25d10a96b5484d2593ac8a45599db6709e5dc4b87: Status 404 returned error can't find the container with id 63c1fd9259d48d8294fb91f25d10a96b5484d2593ac8a45599db6709e5dc4b87 Oct 30 12:54:44.068803 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:44.068769 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l" event={"ID":"b6f03757-55c6-41d8-888a-4ccb9c9d9981","Type":"ContainerStarted","Data":"63c1fd9259d48d8294fb91f25d10a96b5484d2593ac8a45599db6709e5dc4b87"} Oct 30 12:54:50.093240 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:50.093208 4503 generic.go:356] "Generic (PLEG): container finished" podID="b6f03757-55c6-41d8-888a-4ccb9c9d9981" containerID="cb67b0b4c4bd0b8a0cf44dd82e77ab49286f9646917ccb9ffffb7ea6abd6cd79" exitCode=0 Oct 30 12:54:50.093648 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:50.093296 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l" event={"ID":"b6f03757-55c6-41d8-888a-4ccb9c9d9981","Type":"ContainerDied","Data":"cb67b0b4c4bd0b8a0cf44dd82e77ab49286f9646917ccb9ffffb7ea6abd6cd79"} Oct 30 12:54:52.102228 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:52.102189 4503 generic.go:356] "Generic (PLEG): container finished" podID="b6f03757-55c6-41d8-888a-4ccb9c9d9981" containerID="f3b2936ee27075cc611caf335b866aa207ba7068ce7070dcb6a15d5206601172" exitCode=0 Oct 30 12:54:52.102599 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:52.102266 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l" event={"ID":"b6f03757-55c6-41d8-888a-4ccb9c9d9981","Type":"ContainerDied","Data":"f3b2936ee27075cc611caf335b866aa207ba7068ce7070dcb6a15d5206601172"} Oct 30 12:54:58.127978 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:58.127919 4503 generic.go:356] "Generic (PLEG): container finished" podID="b6f03757-55c6-41d8-888a-4ccb9c9d9981" containerID="1c8e18d813cc897db1c1d499b3855c86572c26eac568c3a149f9057df7c2a9e2" exitCode=0 Oct 30 12:54:58.128357 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:58.127995 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l" event={"ID":"b6f03757-55c6-41d8-888a-4ccb9c9d9981","Type":"ContainerDied","Data":"1c8e18d813cc897db1c1d499b3855c86572c26eac568c3a149f9057df7c2a9e2"} Oct 30 12:54:59.254772 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:59.254748 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l" Oct 30 12:54:59.355521 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:59.355481 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6f03757-55c6-41d8-888a-4ccb9c9d9981-kube-api-access-ntt6f\" (UniqueName: \"kubernetes.io/projected/b6f03757-55c6-41d8-888a-4ccb9c9d9981-kube-api-access-ntt6f\") pod \"b6f03757-55c6-41d8-888a-4ccb9c9d9981\" (UID: \"b6f03757-55c6-41d8-888a-4ccb9c9d9981\") " Oct 30 12:54:59.355693 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:59.355548 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6f03757-55c6-41d8-888a-4ccb9c9d9981-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6f03757-55c6-41d8-888a-4ccb9c9d9981-bundle\") pod \"b6f03757-55c6-41d8-888a-4ccb9c9d9981\" (UID: \"b6f03757-55c6-41d8-888a-4ccb9c9d9981\") " Oct 30 12:54:59.355693 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:59.355586 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6f03757-55c6-41d8-888a-4ccb9c9d9981-util\" (UniqueName: \"kubernetes.io/empty-dir/b6f03757-55c6-41d8-888a-4ccb9c9d9981-util\") pod \"b6f03757-55c6-41d8-888a-4ccb9c9d9981\" (UID: \"b6f03757-55c6-41d8-888a-4ccb9c9d9981\") " Oct 30 12:54:59.356256 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:59.356224 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f03757-55c6-41d8-888a-4ccb9c9d9981-bundle" pod "b6f03757-55c6-41d8-888a-4ccb9c9d9981" (UID: "b6f03757-55c6-41d8-888a-4ccb9c9d9981"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:54:59.357779 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:59.357754 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f03757-55c6-41d8-888a-4ccb9c9d9981-kube-api-access-ntt6f" pod "b6f03757-55c6-41d8-888a-4ccb9c9d9981" (UID: "b6f03757-55c6-41d8-888a-4ccb9c9d9981"). InnerVolumeSpecName "kube-api-access-ntt6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:59.360518 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:59.360495 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f03757-55c6-41d8-888a-4ccb9c9d9981-util" pod "b6f03757-55c6-41d8-888a-4ccb9c9d9981" (UID: "b6f03757-55c6-41d8-888a-4ccb9c9d9981"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:54:59.456340 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:59.456299 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntt6f\" (UniqueName: \"kubernetes.io/projected/b6f03757-55c6-41d8-888a-4ccb9c9d9981-kube-api-access-ntt6f\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:54:59.456340 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:59.456332 4503 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6f03757-55c6-41d8-888a-4ccb9c9d9981-bundle\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:54:59.456340 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:54:59.456342 4503 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6f03757-55c6-41d8-888a-4ccb9c9d9981-util\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:54:59.702278 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:54:59.702239 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 12:55:00.136703 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:00.136670 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l" event={"ID":"b6f03757-55c6-41d8-888a-4ccb9c9d9981","Type":"ContainerDied","Data":"63c1fd9259d48d8294fb91f25d10a96b5484d2593ac8a45599db6709e5dc4b87"} Oct 30 12:55:00.136703 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:00.136702 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmtk5l" Oct 30 12:55:00.136897 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:00.136716 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63c1fd9259d48d8294fb91f25d10a96b5484d2593ac8a45599db6709e5dc4b87" Oct 30 12:55:06.481150 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.481118 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-98kpv"] Oct 30 12:55:06.481748 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.481397 4503 cpu_manager.go:414] "containerMap: removing container" podUID="b6f03757-55c6-41d8-888a-4ccb9c9d9981" containerName="pull" Oct 30 12:55:06.481748 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.481407 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f03757-55c6-41d8-888a-4ccb9c9d9981" containerName="pull" Oct 30 12:55:06.481748 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.481423 4503 cpu_manager.go:414] "containerMap: removing container" podUID="b6f03757-55c6-41d8-888a-4ccb9c9d9981" containerName="extract" Oct 30 12:55:06.481748 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.481428 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f03757-55c6-41d8-888a-4ccb9c9d9981" containerName="extract" Oct 30 12:55:06.481748 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.481440 4503 cpu_manager.go:414] "containerMap: removing container" podUID="b6f03757-55c6-41d8-888a-4ccb9c9d9981" containerName="util" Oct 30 12:55:06.481748 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.481445 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f03757-55c6-41d8-888a-4ccb9c9d9981" containerName="util" Oct 30 12:55:06.481748 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.481497 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f03757-55c6-41d8-888a-4ccb9c9d9981" containerName="extract" Oct 30 12:55:06.512964 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.512904 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-98kpv"] Oct 30 12:55:06.513103 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.513037 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-98kpv" Oct 30 12:55:06.530441 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.530417 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 12:55:06.530441 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.530420 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:55:06.530598 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.530456 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vlvlt\"" Oct 30 12:55:06.613914 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.613875 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2k7z\" (UniqueName: \"kubernetes.io/projected/fdf8160f-711b-432e-a753-580082f2f6fa-kube-api-access-w2k7z\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-98kpv\" (UID: \"fdf8160f-711b-432e-a753-580082f2f6fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-98kpv" Oct 30 12:55:06.614096 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.613923 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdf8160f-711b-432e-a753-580082f2f6fa-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-98kpv\" (UID: \"fdf8160f-711b-432e-a753-580082f2f6fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-98kpv" Oct 30 12:55:06.715024 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.714955 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2k7z\" (UniqueName: \"kubernetes.io/projected/fdf8160f-711b-432e-a753-580082f2f6fa-kube-api-access-w2k7z\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-98kpv\" (UID: \"fdf8160f-711b-432e-a753-580082f2f6fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-98kpv" Oct 30 12:55:06.715190 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.715059 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdf8160f-711b-432e-a753-580082f2f6fa-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-98kpv\" (UID: \"fdf8160f-711b-432e-a753-580082f2f6fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-98kpv" Oct 30 12:55:06.715472 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.715455 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdf8160f-711b-432e-a753-580082f2f6fa-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-98kpv\" (UID: \"fdf8160f-711b-432e-a753-580082f2f6fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-98kpv" Oct 30 12:55:06.740325 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.740262 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2k7z\" (UniqueName: \"kubernetes.io/projected/fdf8160f-711b-432e-a753-580082f2f6fa-kube-api-access-w2k7z\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-98kpv\" (UID: \"fdf8160f-711b-432e-a753-580082f2f6fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-98kpv" Oct 30 12:55:06.836675 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:06.836638 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-98kpv" Oct 30 12:55:07.182000 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:07.181972 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-98kpv"] Oct 30 12:55:07.182712 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:55:07.182688 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf8160f_711b_432e_a753_580082f2f6fa.slice/crio-bd0080831dc784ca871d797469b1639b1b56128b5b831efc239894c87e73134d WatchSource:0}: Error finding container bd0080831dc784ca871d797469b1639b1b56128b5b831efc239894c87e73134d: Status 404 returned error can't find the container with id bd0080831dc784ca871d797469b1639b1b56128b5b831efc239894c87e73134d Oct 30 12:55:08.167264 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:08.167225 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-98kpv" event={"ID":"fdf8160f-711b-432e-a753-580082f2f6fa","Type":"ContainerStarted","Data":"bd0080831dc784ca871d797469b1639b1b56128b5b831efc239894c87e73134d"} Oct 30 12:55:10.175913 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:10.175876 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-98kpv" event={"ID":"fdf8160f-711b-432e-a753-580082f2f6fa","Type":"ContainerStarted","Data":"f2039599db02d7ce1afe9161c21557509939105006bef2256e8d4571fcab17e2"} Oct 30 12:55:10.256549 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:10.256493 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-98kpv" podStartSLOduration=2.333092991 podStartE2EDuration="4.256477206s" totalImagesPullingTime="1.923384215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:10.243139679 +0000 UTC m=+204.066367983" watchObservedRunningTime="2026-10-30 12:55:10.256477206 +0000 UTC m=+204.079705547" Oct 30 12:55:11.164606 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.164573 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw"] Oct 30 12:55:11.186216 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.186185 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw" Oct 30 12:55:11.188848 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.188823 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw"] Oct 30 12:55:11.215417 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.215394 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 12:55:11.215576 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.215429 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 12:55:11.215631 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.215618 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-t7t76\"" Oct 30 12:55:11.255978 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.255917 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw\" (UID: \"837cc3e6-df61-410f-bacb-4f5c3c5c9f27\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw" Oct 30 12:55:11.256163 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.256045 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw\" (UID: \"837cc3e6-df61-410f-bacb-4f5c3c5c9f27\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw" Oct 30 12:55:11.256163 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.256094 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24sfm\" (UniqueName: \"kubernetes.io/projected/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-kube-api-access-24sfm\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw\" (UID: \"837cc3e6-df61-410f-bacb-4f5c3c5c9f27\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw" Oct 30 12:55:11.356957 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.356896 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw\" (UID: \"837cc3e6-df61-410f-bacb-4f5c3c5c9f27\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw" Oct 30 12:55:11.357118 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.356964 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw\" (UID: \"837cc3e6-df61-410f-bacb-4f5c3c5c9f27\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw" Oct 30 12:55:11.357118 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.357010 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24sfm\" (UniqueName: \"kubernetes.io/projected/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-kube-api-access-24sfm\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw\" (UID: \"837cc3e6-df61-410f-bacb-4f5c3c5c9f27\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw" Oct 30 12:55:11.357308 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.357288 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw\" (UID: \"837cc3e6-df61-410f-bacb-4f5c3c5c9f27\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw" Oct 30 12:55:11.357342 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.357313 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw\" (UID: \"837cc3e6-df61-410f-bacb-4f5c3c5c9f27\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw" Oct 30 12:55:11.382562 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.382530 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24sfm\" (UniqueName: \"kubernetes.io/projected/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-kube-api-access-24sfm\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw\" (UID: \"837cc3e6-df61-410f-bacb-4f5c3c5c9f27\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw" Oct 30 12:55:11.492798 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.492710 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-z4psz"] Oct 30 12:55:11.509732 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.509707 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4psz" Oct 30 12:55:11.519964 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.519921 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-z4psz"] Oct 30 12:55:11.526528 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.526499 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 12:55:11.526729 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.526707 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-lr4dj\"" Oct 30 12:55:11.526799 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.526761 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 12:55:11.536468 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.536442 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw" Oct 30 12:55:11.659517 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.659482 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcf2e6e9-d1d9-46d2-8d66-75ffa56cf603-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-z4psz\" (UID: \"fcf2e6e9-d1d9-46d2-8d66-75ffa56cf603\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4psz" Oct 30 12:55:11.659700 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.659556 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cq8w\" (UniqueName: \"kubernetes.io/projected/fcf2e6e9-d1d9-46d2-8d66-75ffa56cf603-kube-api-access-4cq8w\") pod \"cert-manager-webhook-5dbbbf75cc-z4psz\" (UID: \"fcf2e6e9-d1d9-46d2-8d66-75ffa56cf603\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4psz" Oct 30 12:55:11.760319 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.760224 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cq8w\" (UniqueName: \"kubernetes.io/projected/fcf2e6e9-d1d9-46d2-8d66-75ffa56cf603-kube-api-access-4cq8w\") pod \"cert-manager-webhook-5dbbbf75cc-z4psz\" (UID: \"fcf2e6e9-d1d9-46d2-8d66-75ffa56cf603\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4psz" Oct 30 12:55:11.760319 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.760303 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcf2e6e9-d1d9-46d2-8d66-75ffa56cf603-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-z4psz\" (UID: \"fcf2e6e9-d1d9-46d2-8d66-75ffa56cf603\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4psz" Oct 30 12:55:11.785452 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.785418 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcf2e6e9-d1d9-46d2-8d66-75ffa56cf603-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-z4psz\" (UID: \"fcf2e6e9-d1d9-46d2-8d66-75ffa56cf603\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4psz" Oct 30 12:55:11.785647 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.785628 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cq8w\" (UniqueName: \"kubernetes.io/projected/fcf2e6e9-d1d9-46d2-8d66-75ffa56cf603-kube-api-access-4cq8w\") pod \"cert-manager-webhook-5dbbbf75cc-z4psz\" (UID: \"fcf2e6e9-d1d9-46d2-8d66-75ffa56cf603\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4psz" Oct 30 12:55:11.845180 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.845142 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4psz" Oct 30 12:55:11.858849 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:11.858821 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw"] Oct 30 12:55:11.860017 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:55:11.859992 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837cc3e6_df61_410f_bacb_4f5c3c5c9f27.slice/crio-508b5908a15d2ef29f028d10b01693aea35b3e0fccdf4bba18582e9bfd48e2cc WatchSource:0}: Error finding container 508b5908a15d2ef29f028d10b01693aea35b3e0fccdf4bba18582e9bfd48e2cc: Status 404 returned error can't find the container with id 508b5908a15d2ef29f028d10b01693aea35b3e0fccdf4bba18582e9bfd48e2cc Oct 30 12:55:12.171241 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:12.171217 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-z4psz"] Oct 30 12:55:12.172437 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:55:12.172410 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcf2e6e9_d1d9_46d2_8d66_75ffa56cf603.slice/crio-fb0aa76b9d13271ca6d10b449ccf3b88b0506a149df089cca7f1aa7452abd479 WatchSource:0}: Error finding container fb0aa76b9d13271ca6d10b449ccf3b88b0506a149df089cca7f1aa7452abd479: Status 404 returned error can't find the container with id fb0aa76b9d13271ca6d10b449ccf3b88b0506a149df089cca7f1aa7452abd479 Oct 30 12:55:12.184572 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:12.184545 4503 generic.go:356] "Generic (PLEG): container finished" podID="837cc3e6-df61-410f-bacb-4f5c3c5c9f27" containerID="83988cda125ee066d0c52b8f9d6408340a1dd0a4e1995d5c6fa9d28f66ce9f02" exitCode=0 Oct 30 12:55:12.184664 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:12.184628 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw" event={"ID":"837cc3e6-df61-410f-bacb-4f5c3c5c9f27","Type":"ContainerDied","Data":"83988cda125ee066d0c52b8f9d6408340a1dd0a4e1995d5c6fa9d28f66ce9f02"} Oct 30 12:55:12.184725 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:12.184706 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw" event={"ID":"837cc3e6-df61-410f-bacb-4f5c3c5c9f27","Type":"ContainerStarted","Data":"508b5908a15d2ef29f028d10b01693aea35b3e0fccdf4bba18582e9bfd48e2cc"} Oct 30 12:55:12.185893 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:12.185871 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4psz" event={"ID":"fcf2e6e9-d1d9-46d2-8d66-75ffa56cf603","Type":"ContainerStarted","Data":"fb0aa76b9d13271ca6d10b449ccf3b88b0506a149df089cca7f1aa7452abd479"} Oct 30 12:55:15.200679 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:15.200637 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4psz" event={"ID":"fcf2e6e9-d1d9-46d2-8d66-75ffa56cf603","Type":"ContainerStarted","Data":"879496b3f9f67d6ccc8b4c3bbf3c601e88ec49a9e79f9f1b64dc22666f85b486"} Oct 30 12:55:15.201169 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:15.200696 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4psz" Oct 30 12:55:15.202312 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:15.202286 4503 generic.go:356] "Generic (PLEG): container finished" podID="837cc3e6-df61-410f-bacb-4f5c3c5c9f27" containerID="8665421d245cbe69003345616b15ecebdb23f643a04c39051d75b70d3f419df4" exitCode=0 Oct 30 12:55:15.202438 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:15.202333 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw" event={"ID":"837cc3e6-df61-410f-bacb-4f5c3c5c9f27","Type":"ContainerDied","Data":"8665421d245cbe69003345616b15ecebdb23f643a04c39051d75b70d3f419df4"} Oct 30 12:55:15.243194 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:15.243149 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4psz" podStartSLOduration=1.6704509 podStartE2EDuration="4.243135833s" totalImagesPullingTime="2.572684933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:15.242720174 +0000 UTC m=+209.065948451" watchObservedRunningTime="2026-10-30 12:55:15.243135833 +0000 UTC m=+209.066364109" Oct 30 12:55:16.208798 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:16.208764 4503 generic.go:356] "Generic (PLEG): container finished" podID="837cc3e6-df61-410f-bacb-4f5c3c5c9f27" containerID="3dc129edb389cf2f5555a5462f57d6e24442a6bcbf06a915aa89fab9ed151e4c" exitCode=0 Oct 30 12:55:16.209300 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:16.208835 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw" event={"ID":"837cc3e6-df61-410f-bacb-4f5c3c5c9f27","Type":"ContainerDied","Data":"3dc129edb389cf2f5555a5462f57d6e24442a6bcbf06a915aa89fab9ed151e4c"} Oct 30 12:55:17.330879 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:17.330856 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw" Oct 30 12:55:17.406786 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:17.406759 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-bundle\" (UniqueName: \"kubernetes.io/empty-dir/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-bundle\") pod \"837cc3e6-df61-410f-bacb-4f5c3c5c9f27\" (UID: \"837cc3e6-df61-410f-bacb-4f5c3c5c9f27\") " Oct 30 12:55:17.406972 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:17.406811 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-kube-api-access-24sfm\" (UniqueName: \"kubernetes.io/projected/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-kube-api-access-24sfm\") pod \"837cc3e6-df61-410f-bacb-4f5c3c5c9f27\" (UID: \"837cc3e6-df61-410f-bacb-4f5c3c5c9f27\") " Oct 30 12:55:17.406972 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:17.406855 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-util\" (UniqueName: \"kubernetes.io/empty-dir/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-util\") pod \"837cc3e6-df61-410f-bacb-4f5c3c5c9f27\" (UID: \"837cc3e6-df61-410f-bacb-4f5c3c5c9f27\") " Oct 30 12:55:17.407250 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:17.407221 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-bundle" pod "837cc3e6-df61-410f-bacb-4f5c3c5c9f27" (UID: "837cc3e6-df61-410f-bacb-4f5c3c5c9f27"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:55:17.409020 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:17.409000 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-kube-api-access-24sfm" pod "837cc3e6-df61-410f-bacb-4f5c3c5c9f27" (UID: "837cc3e6-df61-410f-bacb-4f5c3c5c9f27"). InnerVolumeSpecName "kube-api-access-24sfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:55:17.411460 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:17.411426 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-util" pod "837cc3e6-df61-410f-bacb-4f5c3c5c9f27" (UID: "837cc3e6-df61-410f-bacb-4f5c3c5c9f27"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:55:17.508243 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:17.508159 4503 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-bundle\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:55:17.508243 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:17.508191 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24sfm\" (UniqueName: \"kubernetes.io/projected/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-kube-api-access-24sfm\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:55:17.508243 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:17.508202 4503 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/837cc3e6-df61-410f-bacb-4f5c3c5c9f27-util\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:55:18.218441 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:18.218405 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw" event={"ID":"837cc3e6-df61-410f-bacb-4f5c3c5c9f27","Type":"ContainerDied","Data":"508b5908a15d2ef29f028d10b01693aea35b3e0fccdf4bba18582e9bfd48e2cc"} Oct 30 12:55:18.218441 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:18.218427 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mzwmw" Oct 30 12:55:18.218656 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:18.218458 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="508b5908a15d2ef29f028d10b01693aea35b3e0fccdf4bba18582e9bfd48e2cc" Oct 30 12:55:21.211505 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:21.211471 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4psz" Oct 30 12:55:28.370865 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.370835 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-x5thv"] Oct 30 12:55:28.371284 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.371136 4503 cpu_manager.go:414] "containerMap: removing container" podUID="837cc3e6-df61-410f-bacb-4f5c3c5c9f27" containerName="pull" Oct 30 12:55:28.371284 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.371147 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="837cc3e6-df61-410f-bacb-4f5c3c5c9f27" containerName="pull" Oct 30 12:55:28.371284 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.371158 4503 cpu_manager.go:414] "containerMap: removing container" podUID="837cc3e6-df61-410f-bacb-4f5c3c5c9f27" containerName="extract" Oct 30 12:55:28.371284 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.371163 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="837cc3e6-df61-410f-bacb-4f5c3c5c9f27" containerName="extract" Oct 30 12:55:28.371284 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.371173 4503 cpu_manager.go:414] "containerMap: removing container" podUID="837cc3e6-df61-410f-bacb-4f5c3c5c9f27" containerName="util" Oct 30 12:55:28.371284 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.371178 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="837cc3e6-df61-410f-bacb-4f5c3c5c9f27" containerName="util" Oct 30 12:55:28.371284 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.371227 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="837cc3e6-df61-410f-bacb-4f5c3c5c9f27" containerName="extract" Oct 30 12:55:28.373118 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.373101 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-x5thv" Oct 30 12:55:28.402144 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.402117 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-9489w\"" Oct 30 12:55:28.402144 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.402115 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:55:28.402144 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.402153 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 12:55:28.403992 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.403971 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-x5thv"] Oct 30 12:55:28.499112 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.499082 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8333516c-4351-452f-b2c3-6919fde319af-tmp\") pod \"openshift-lws-operator-8465994787-x5thv\" (UID: \"8333516c-4351-452f-b2c3-6919fde319af\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-x5thv" Oct 30 12:55:28.499112 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.499115 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5hch\" (UniqueName: \"kubernetes.io/projected/8333516c-4351-452f-b2c3-6919fde319af-kube-api-access-r5hch\") pod \"openshift-lws-operator-8465994787-x5thv\" (UID: \"8333516c-4351-452f-b2c3-6919fde319af\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-x5thv" Oct 30 12:55:28.599612 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.599574 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8333516c-4351-452f-b2c3-6919fde319af-tmp\") pod \"openshift-lws-operator-8465994787-x5thv\" (UID: \"8333516c-4351-452f-b2c3-6919fde319af\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-x5thv" Oct 30 12:55:28.599612 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.599615 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5hch\" (UniqueName: \"kubernetes.io/projected/8333516c-4351-452f-b2c3-6919fde319af-kube-api-access-r5hch\") pod \"openshift-lws-operator-8465994787-x5thv\" (UID: \"8333516c-4351-452f-b2c3-6919fde319af\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-x5thv" Oct 30 12:55:28.600051 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.600030 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8333516c-4351-452f-b2c3-6919fde319af-tmp\") pod \"openshift-lws-operator-8465994787-x5thv\" (UID: \"8333516c-4351-452f-b2c3-6919fde319af\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-x5thv" Oct 30 12:55:28.624841 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.624780 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5hch\" (UniqueName: \"kubernetes.io/projected/8333516c-4351-452f-b2c3-6919fde319af-kube-api-access-r5hch\") pod \"openshift-lws-operator-8465994787-x5thv\" (UID: \"8333516c-4351-452f-b2c3-6919fde319af\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-x5thv" Oct 30 12:55:28.683685 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.683649 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-x5thv" Oct 30 12:55:28.990596 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:28.985297 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-x5thv"] Oct 30 12:55:29.261794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:29.261715 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-x5thv" event={"ID":"8333516c-4351-452f-b2c3-6919fde319af","Type":"ContainerStarted","Data":"ab7fce0ddbce6689ccd3416d3c81811c3b8a07d50146f6edf1fc7c878222666a"} Oct 30 12:55:29.706899 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:55:29.706848 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 12:55:31.272532 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:31.272499 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-x5thv" event={"ID":"8333516c-4351-452f-b2c3-6919fde319af","Type":"ContainerStarted","Data":"01f65067e20299dbdf4dab594f259683ea02be3e67a6fbc5c001e61e82bc277f"} Oct 30 12:55:31.315585 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:31.315532 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-x5thv" podStartSLOduration=1.9215000340000001 podStartE2EDuration="3.315516968s" totalImagesPullingTime="1.394016934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:31.314556662 +0000 UTC m=+225.137784950" watchObservedRunningTime="2026-10-30 12:55:31.315516968 +0000 UTC m=+225.138745245" Oct 30 12:55:36.072322 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:36.072285 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv"] Oct 30 12:55:36.074789 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:36.074771 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv" Oct 30 12:55:36.103238 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:36.103213 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-z6rn2\"" Oct 30 12:55:36.103619 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:36.103604 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 12:55:36.103672 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:36.103627 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 12:55:36.104884 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:36.104864 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv"] Oct 30 12:55:36.262462 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:36.262427 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv\" (UID: \"3527aa38-3734-4bc9-b7b3-55add8c5dd4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv" Oct 30 12:55:36.262628 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:36.262475 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv\" (UID: \"3527aa38-3734-4bc9-b7b3-55add8c5dd4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv" Oct 30 12:55:36.262628 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:36.262505 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gf8g\" (UniqueName: \"kubernetes.io/projected/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-kube-api-access-8gf8g\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv\" (UID: \"3527aa38-3734-4bc9-b7b3-55add8c5dd4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv" Oct 30 12:55:36.363440 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:36.363360 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv\" (UID: \"3527aa38-3734-4bc9-b7b3-55add8c5dd4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv" Oct 30 12:55:36.363440 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:36.363415 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv\" (UID: \"3527aa38-3734-4bc9-b7b3-55add8c5dd4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv" Oct 30 12:55:36.363619 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:36.363602 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gf8g\" (UniqueName: \"kubernetes.io/projected/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-kube-api-access-8gf8g\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv\" (UID: \"3527aa38-3734-4bc9-b7b3-55add8c5dd4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv" Oct 30 12:55:36.363858 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:36.363838 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv\" (UID: \"3527aa38-3734-4bc9-b7b3-55add8c5dd4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv" Oct 30 12:55:36.363910 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:36.363891 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv\" (UID: \"3527aa38-3734-4bc9-b7b3-55add8c5dd4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv" Oct 30 12:55:36.389051 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:36.389025 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gf8g\" (UniqueName: \"kubernetes.io/projected/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-kube-api-access-8gf8g\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv\" (UID: \"3527aa38-3734-4bc9-b7b3-55add8c5dd4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv" Oct 30 12:55:36.687705 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:36.687673 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv" Oct 30 12:55:36.983655 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:36.983622 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv"] Oct 30 12:55:36.983975 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:55:36.983916 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3527aa38_3734_4bc9_b7b3_55add8c5dd4a.slice/crio-597edb4f2720d887eff1bfa07fa81819645bbfdcf7abade1b1991c6d1fd32de9 WatchSource:0}: Error finding container 597edb4f2720d887eff1bfa07fa81819645bbfdcf7abade1b1991c6d1fd32de9: Status 404 returned error can't find the container with id 597edb4f2720d887eff1bfa07fa81819645bbfdcf7abade1b1991c6d1fd32de9 Oct 30 12:55:37.301191 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:37.301109 4503 generic.go:356] "Generic (PLEG): container finished" podID="3527aa38-3734-4bc9-b7b3-55add8c5dd4a" containerID="845fde3f7f88fec23d5a432a00e88f81e46027b9336ee1359721efa878906a19" exitCode=0 Oct 30 12:55:37.301619 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:37.301196 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv" event={"ID":"3527aa38-3734-4bc9-b7b3-55add8c5dd4a","Type":"ContainerDied","Data":"845fde3f7f88fec23d5a432a00e88f81e46027b9336ee1359721efa878906a19"} Oct 30 12:55:37.301619 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:37.301247 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv" event={"ID":"3527aa38-3734-4bc9-b7b3-55add8c5dd4a","Type":"ContainerStarted","Data":"597edb4f2720d887eff1bfa07fa81819645bbfdcf7abade1b1991c6d1fd32de9"} Oct 30 12:55:38.307276 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:38.307241 4503 generic.go:356] "Generic (PLEG): container finished" podID="3527aa38-3734-4bc9-b7b3-55add8c5dd4a" containerID="942f005abc1e7bffffe8e45cb548da61b6138037f1abd5e2098c6f02848a322f" exitCode=0 Oct 30 12:55:38.307735 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:38.307328 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv" event={"ID":"3527aa38-3734-4bc9-b7b3-55add8c5dd4a","Type":"ContainerDied","Data":"942f005abc1e7bffffe8e45cb548da61b6138037f1abd5e2098c6f02848a322f"} Oct 30 12:55:39.313157 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:39.313119 4503 generic.go:356] "Generic (PLEG): container finished" podID="3527aa38-3734-4bc9-b7b3-55add8c5dd4a" containerID="3871f0156f4ca7480449a50dc2ca84f268b2a64eda1f70e449146dcd6bbbb2bc" exitCode=0 Oct 30 12:55:39.313531 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:39.313176 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv" event={"ID":"3527aa38-3734-4bc9-b7b3-55add8c5dd4a","Type":"ContainerDied","Data":"3871f0156f4ca7480449a50dc2ca84f268b2a64eda1f70e449146dcd6bbbb2bc"} Oct 30 12:55:40.438236 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:40.438214 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv" Oct 30 12:55:40.495535 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:40.495503 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-kube-api-access-8gf8g\" (UniqueName: \"kubernetes.io/projected/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-kube-api-access-8gf8g\") pod \"3527aa38-3734-4bc9-b7b3-55add8c5dd4a\" (UID: \"3527aa38-3734-4bc9-b7b3-55add8c5dd4a\") " Oct 30 12:55:40.495686 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:40.495594 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-util\" (UniqueName: \"kubernetes.io/empty-dir/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-util\") pod \"3527aa38-3734-4bc9-b7b3-55add8c5dd4a\" (UID: \"3527aa38-3734-4bc9-b7b3-55add8c5dd4a\") " Oct 30 12:55:40.495686 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:40.495625 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-bundle\") pod \"3527aa38-3734-4bc9-b7b3-55add8c5dd4a\" (UID: \"3527aa38-3734-4bc9-b7b3-55add8c5dd4a\") " Oct 30 12:55:40.496236 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:40.496205 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-bundle" pod "3527aa38-3734-4bc9-b7b3-55add8c5dd4a" (UID: "3527aa38-3734-4bc9-b7b3-55add8c5dd4a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:55:40.497701 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:40.497670 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-kube-api-access-8gf8g" pod "3527aa38-3734-4bc9-b7b3-55add8c5dd4a" (UID: "3527aa38-3734-4bc9-b7b3-55add8c5dd4a"). InnerVolumeSpecName "kube-api-access-8gf8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:55:40.504653 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:40.504607 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-util" pod "3527aa38-3734-4bc9-b7b3-55add8c5dd4a" (UID: "3527aa38-3734-4bc9-b7b3-55add8c5dd4a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:55:40.596791 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:40.596707 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gf8g\" (UniqueName: \"kubernetes.io/projected/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-kube-api-access-8gf8g\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:55:40.596791 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:40.596747 4503 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-util\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:55:40.596791 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:40.596760 4503 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3527aa38-3734-4bc9-b7b3-55add8c5dd4a-bundle\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:55:41.322677 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:41.322638 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv" event={"ID":"3527aa38-3734-4bc9-b7b3-55add8c5dd4a","Type":"ContainerDied","Data":"597edb4f2720d887eff1bfa07fa81819645bbfdcf7abade1b1991c6d1fd32de9"} Oct 30 12:55:41.322878 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:41.322686 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x5glv" Oct 30 12:55:41.322878 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:41.322709 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="597edb4f2720d887eff1bfa07fa81819645bbfdcf7abade1b1991c6d1fd32de9" Oct 30 12:55:45.937564 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:45.937532 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25"] Oct 30 12:55:45.937974 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:45.937828 4503 cpu_manager.go:414] "containerMap: removing container" podUID="3527aa38-3734-4bc9-b7b3-55add8c5dd4a" containerName="extract" Oct 30 12:55:45.937974 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:45.937838 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3527aa38-3734-4bc9-b7b3-55add8c5dd4a" containerName="extract" Oct 30 12:55:45.937974 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:45.937853 4503 cpu_manager.go:414] "containerMap: removing container" podUID="3527aa38-3734-4bc9-b7b3-55add8c5dd4a" containerName="pull" Oct 30 12:55:45.937974 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:45.937858 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3527aa38-3734-4bc9-b7b3-55add8c5dd4a" containerName="pull" Oct 30 12:55:45.937974 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:45.937868 4503 cpu_manager.go:414] "containerMap: removing container" podUID="3527aa38-3734-4bc9-b7b3-55add8c5dd4a" containerName="util" Oct 30 12:55:45.937974 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:45.937872 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3527aa38-3734-4bc9-b7b3-55add8c5dd4a" containerName="util" Oct 30 12:55:45.937974 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:45.937928 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="3527aa38-3734-4bc9-b7b3-55add8c5dd4a" containerName="extract" Oct 30 12:55:45.942394 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:45.942375 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:45.957557 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:45.957532 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25"] Oct 30 12:55:45.988110 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:45.988080 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 12:55:45.988216 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:45.988116 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 12:55:45.988330 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:45.988314 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 12:55:45.988401 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:45.988343 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-frp85\"" Oct 30 12:55:45.988401 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:45.988370 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 12:55:45.988401 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:45.988381 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 12:55:46.042428 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.042400 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/34145300-0dc5-48e9-91ea-945034a3e538-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25\" (UID: \"34145300-0dc5-48e9-91ea-945034a3e538\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:46.042562 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.042468 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f9lx\" (UniqueName: \"kubernetes.io/projected/34145300-0dc5-48e9-91ea-945034a3e538-kube-api-access-7f9lx\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25\" (UID: \"34145300-0dc5-48e9-91ea-945034a3e538\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:46.042626 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.042566 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/34145300-0dc5-48e9-91ea-945034a3e538-manifests\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25\" (UID: \"34145300-0dc5-48e9-91ea-945034a3e538\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:46.042626 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.042605 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/34145300-0dc5-48e9-91ea-945034a3e538-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25\" (UID: \"34145300-0dc5-48e9-91ea-945034a3e538\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:46.042754 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.042637 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34145300-0dc5-48e9-91ea-945034a3e538-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25\" (UID: \"34145300-0dc5-48e9-91ea-945034a3e538\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:46.042754 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.042654 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34145300-0dc5-48e9-91ea-945034a3e538-tmp\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25\" (UID: \"34145300-0dc5-48e9-91ea-945034a3e538\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:46.143712 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.143676 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/34145300-0dc5-48e9-91ea-945034a3e538-manifests\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25\" (UID: \"34145300-0dc5-48e9-91ea-945034a3e538\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:46.143910 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.143719 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/34145300-0dc5-48e9-91ea-945034a3e538-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25\" (UID: \"34145300-0dc5-48e9-91ea-945034a3e538\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:46.143910 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.143753 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34145300-0dc5-48e9-91ea-945034a3e538-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25\" (UID: \"34145300-0dc5-48e9-91ea-945034a3e538\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:46.143910 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.143774 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34145300-0dc5-48e9-91ea-945034a3e538-tmp\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25\" (UID: \"34145300-0dc5-48e9-91ea-945034a3e538\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:46.143910 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.143809 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/34145300-0dc5-48e9-91ea-945034a3e538-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25\" (UID: \"34145300-0dc5-48e9-91ea-945034a3e538\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:46.143910 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.143893 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f9lx\" (UniqueName: \"kubernetes.io/projected/34145300-0dc5-48e9-91ea-945034a3e538-kube-api-access-7f9lx\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25\" (UID: \"34145300-0dc5-48e9-91ea-945034a3e538\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:46.144189 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.144172 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/34145300-0dc5-48e9-91ea-945034a3e538-manifests\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25\" (UID: \"34145300-0dc5-48e9-91ea-945034a3e538\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:46.146245 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.146218 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34145300-0dc5-48e9-91ea-945034a3e538-tmp\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25\" (UID: \"34145300-0dc5-48e9-91ea-945034a3e538\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:46.146361 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.146279 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/34145300-0dc5-48e9-91ea-945034a3e538-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25\" (UID: \"34145300-0dc5-48e9-91ea-945034a3e538\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:46.146361 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.146310 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34145300-0dc5-48e9-91ea-945034a3e538-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25\" (UID: \"34145300-0dc5-48e9-91ea-945034a3e538\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:46.146460 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.146437 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/34145300-0dc5-48e9-91ea-945034a3e538-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25\" (UID: \"34145300-0dc5-48e9-91ea-945034a3e538\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:46.169527 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.169502 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f9lx\" (UniqueName: \"kubernetes.io/projected/34145300-0dc5-48e9-91ea-945034a3e538-kube-api-access-7f9lx\") pod \"opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25\" (UID: \"34145300-0dc5-48e9-91ea-945034a3e538\") " pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:46.254843 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.254773 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:46.612672 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:46.612645 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25"] Oct 30 12:55:46.613655 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:55:46.613628 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34145300_0dc5_48e9_91ea_945034a3e538.slice/crio-97e3a137ba6fbfb136ee714aba98efabba73f3c00d9dbda7573c0e925be73af2 WatchSource:0}: Error finding container 97e3a137ba6fbfb136ee714aba98efabba73f3c00d9dbda7573c0e925be73af2: Status 404 returned error can't find the container with id 97e3a137ba6fbfb136ee714aba98efabba73f3c00d9dbda7573c0e925be73af2 Oct 30 12:55:47.353759 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:47.353719 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" event={"ID":"34145300-0dc5-48e9-91ea-945034a3e538","Type":"ContainerStarted","Data":"97e3a137ba6fbfb136ee714aba98efabba73f3c00d9dbda7573c0e925be73af2"} Oct 30 12:55:49.365929 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:49.365902 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" event={"ID":"34145300-0dc5-48e9-91ea-945034a3e538","Type":"ContainerStarted","Data":"dc52f108056266fb3cf8ed6b8131aca20230e37144f5bab471de18f3dd196376"} Oct 30 12:55:50.370837 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:50.370753 4503 generic.go:356] "Generic (PLEG): container finished" podID="34145300-0dc5-48e9-91ea-945034a3e538" containerID="dc52f108056266fb3cf8ed6b8131aca20230e37144f5bab471de18f3dd196376" exitCode=0 Oct 30 12:55:50.370837 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:50.370808 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" event={"ID":"34145300-0dc5-48e9-91ea-945034a3e538","Type":"ContainerDied","Data":"dc52f108056266fb3cf8ed6b8131aca20230e37144f5bab471de18f3dd196376"} Oct 30 12:55:51.376931 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:51.376890 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" event={"ID":"34145300-0dc5-48e9-91ea-945034a3e538","Type":"ContainerStarted","Data":"fe26dfc31c85fd4b993be41e2c0835b63d5c027d7e818c085ca58a93eef55bd9"} Oct 30 12:55:51.377350 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:51.376990 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:55:51.468074 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:55:51.468021 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" podStartSLOduration=2.840802456 podStartE2EDuration="6.467997845s" totalImagesPullingTime="3.045271566s" totalInitContainerRuntime="581.923823ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:51.455059307 +0000 UTC m=+245.278287585" watchObservedRunningTime="2026-10-30 12:55:51.467997845 +0000 UTC m=+245.291226121" Oct 30 12:55:59.699431 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:55:59.699386 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 12:56:02.383923 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:02.383890 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25" Oct 30 12:56:08.955491 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:08.955450 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-qlkgt"] Oct 30 12:56:08.958721 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:08.958704 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-qlkgt" Oct 30 12:56:08.975183 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:08.975166 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\"" Oct 30 12:56:08.987128 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:08.987096 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-q8m8k\"" Oct 30 12:56:08.988546 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:08.988522 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-qlkgt"] Oct 30 12:56:09.035361 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:09.035324 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/ec8777f0-af67-4906-8537-631e712dd6b6-webhook-certs\") pod \"odh-observability-584f99c788-qlkgt\" (UID: \"ec8777f0-af67-4906-8537-631e712dd6b6\") " pod="opendatahub/odh-observability-584f99c788-qlkgt" Oct 30 12:56:09.035521 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:09.035395 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzwdh\" (UniqueName: \"kubernetes.io/projected/ec8777f0-af67-4906-8537-631e712dd6b6-kube-api-access-mzwdh\") pod \"odh-observability-584f99c788-qlkgt\" (UID: \"ec8777f0-af67-4906-8537-631e712dd6b6\") " pod="opendatahub/odh-observability-584f99c788-qlkgt" Oct 30 12:56:09.136298 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:09.136260 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzwdh\" (UniqueName: \"kubernetes.io/projected/ec8777f0-af67-4906-8537-631e712dd6b6-kube-api-access-mzwdh\") pod \"odh-observability-584f99c788-qlkgt\" (UID: \"ec8777f0-af67-4906-8537-631e712dd6b6\") " pod="opendatahub/odh-observability-584f99c788-qlkgt" Oct 30 12:56:09.136468 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:09.136340 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/ec8777f0-af67-4906-8537-631e712dd6b6-webhook-certs\") pod \"odh-observability-584f99c788-qlkgt\" (UID: \"ec8777f0-af67-4906-8537-631e712dd6b6\") " pod="opendatahub/odh-observability-584f99c788-qlkgt" Oct 30 12:56:09.136468 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:56:09.136444 4503 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 30 12:56:09.136554 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:56:09.136508 4503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec8777f0-af67-4906-8537-631e712dd6b6-webhook-certs podName:ec8777f0-af67-4906-8537-631e712dd6b6 nodeName:}" failed. No retries permitted until 2026-10-30 12:56:09.636489653 +0000 UTC m=+263.459717925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/ec8777f0-af67-4906-8537-631e712dd6b6-webhook-certs") pod "odh-observability-584f99c788-qlkgt" (UID: "ec8777f0-af67-4906-8537-631e712dd6b6") : secret "odh-observability-webhook-cert" not found Oct 30 12:56:09.163091 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:09.163068 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzwdh\" (UniqueName: \"kubernetes.io/projected/ec8777f0-af67-4906-8537-631e712dd6b6-kube-api-access-mzwdh\") pod \"odh-observability-584f99c788-qlkgt\" (UID: \"ec8777f0-af67-4906-8537-631e712dd6b6\") " pod="opendatahub/odh-observability-584f99c788-qlkgt" Oct 30 12:56:09.641806 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:09.641771 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/ec8777f0-af67-4906-8537-631e712dd6b6-webhook-certs\") pod \"odh-observability-584f99c788-qlkgt\" (UID: \"ec8777f0-af67-4906-8537-631e712dd6b6\") " pod="opendatahub/odh-observability-584f99c788-qlkgt" Oct 30 12:56:09.644212 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:09.644178 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/ec8777f0-af67-4906-8537-631e712dd6b6-webhook-certs\") pod \"odh-observability-584f99c788-qlkgt\" (UID: \"ec8777f0-af67-4906-8537-631e712dd6b6\") " pod="opendatahub/odh-observability-584f99c788-qlkgt" Oct 30 12:56:09.870794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:09.870756 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-qlkgt" Oct 30 12:56:10.170427 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:10.170402 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-qlkgt"] Oct 30 12:56:10.171166 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:56:10.171139 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8777f0_af67_4906_8537_631e712dd6b6.slice/crio-e50d4b57abc3f5d95b17fcc99bbd8e3bb8af77c89d62c7e88ff90a59ad38fadd WatchSource:0}: Error finding container e50d4b57abc3f5d95b17fcc99bbd8e3bb8af77c89d62c7e88ff90a59ad38fadd: Status 404 returned error can't find the container with id e50d4b57abc3f5d95b17fcc99bbd8e3bb8af77c89d62c7e88ff90a59ad38fadd Oct 30 12:56:10.457608 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:10.457528 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-qlkgt" event={"ID":"ec8777f0-af67-4906-8537-631e712dd6b6","Type":"ContainerStarted","Data":"e50d4b57abc3f5d95b17fcc99bbd8e3bb8af77c89d62c7e88ff90a59ad38fadd"} Oct 30 12:56:11.250437 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:11.250406 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d59dd5dc7-8dvmm"] Oct 30 12:56:11.253656 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:11.253630 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-8dvmm" Oct 30 12:56:11.284450 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:11.284421 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 12:56:11.284607 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:11.284573 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9fp8r\"" Oct 30 12:56:11.284683 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:11.284605 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 12:56:11.284683 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:11.284635 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 12:56:11.284683 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:11.284644 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 12:56:11.291970 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:11.286927 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d59dd5dc7-8dvmm"] Oct 30 12:56:11.355450 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:11.355408 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da32ea91-80d7-48d5-9167-882b4736108e-tls-certs\") pod \"kube-auth-proxy-d59dd5dc7-8dvmm\" (UID: \"da32ea91-80d7-48d5-9167-882b4736108e\") " pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-8dvmm" Oct 30 12:56:11.355636 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:11.355457 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da32ea91-80d7-48d5-9167-882b4736108e-tmp\") pod \"kube-auth-proxy-d59dd5dc7-8dvmm\" (UID: \"da32ea91-80d7-48d5-9167-882b4736108e\") " pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-8dvmm" Oct 30 12:56:11.355636 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:11.355497 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j29n7\" (UniqueName: \"kubernetes.io/projected/da32ea91-80d7-48d5-9167-882b4736108e-kube-api-access-j29n7\") pod \"kube-auth-proxy-d59dd5dc7-8dvmm\" (UID: \"da32ea91-80d7-48d5-9167-882b4736108e\") " pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-8dvmm" Oct 30 12:56:11.456399 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:11.456365 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da32ea91-80d7-48d5-9167-882b4736108e-tls-certs\") pod \"kube-auth-proxy-d59dd5dc7-8dvmm\" (UID: \"da32ea91-80d7-48d5-9167-882b4736108e\") " pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-8dvmm" Oct 30 12:56:11.456583 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:11.456410 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da32ea91-80d7-48d5-9167-882b4736108e-tmp\") pod \"kube-auth-proxy-d59dd5dc7-8dvmm\" (UID: \"da32ea91-80d7-48d5-9167-882b4736108e\") " pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-8dvmm" Oct 30 12:56:11.456583 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:11.456446 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j29n7\" (UniqueName: \"kubernetes.io/projected/da32ea91-80d7-48d5-9167-882b4736108e-kube-api-access-j29n7\") pod \"kube-auth-proxy-d59dd5dc7-8dvmm\" (UID: \"da32ea91-80d7-48d5-9167-882b4736108e\") " pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-8dvmm" Oct 30 12:56:11.460003 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:11.459974 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da32ea91-80d7-48d5-9167-882b4736108e-tmp\") pod \"kube-auth-proxy-d59dd5dc7-8dvmm\" (UID: \"da32ea91-80d7-48d5-9167-882b4736108e\") " pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-8dvmm" Oct 30 12:56:11.460455 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:11.460423 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da32ea91-80d7-48d5-9167-882b4736108e-tls-certs\") pod \"kube-auth-proxy-d59dd5dc7-8dvmm\" (UID: \"da32ea91-80d7-48d5-9167-882b4736108e\") " pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-8dvmm" Oct 30 12:56:11.481513 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:11.481485 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j29n7\" (UniqueName: \"kubernetes.io/projected/da32ea91-80d7-48d5-9167-882b4736108e-kube-api-access-j29n7\") pod \"kube-auth-proxy-d59dd5dc7-8dvmm\" (UID: \"da32ea91-80d7-48d5-9167-882b4736108e\") " pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-8dvmm" Oct 30 12:56:11.569373 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:11.569289 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-8dvmm" Oct 30 12:56:11.888949 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:11.888910 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d59dd5dc7-8dvmm"] Oct 30 12:56:11.889848 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:56:11.889820 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda32ea91_80d7_48d5_9167_882b4736108e.slice/crio-1b83866927554d2bb21fc6bc1d43484e929aa51a09897e2b26ac20d74ce036be WatchSource:0}: Error finding container 1b83866927554d2bb21fc6bc1d43484e929aa51a09897e2b26ac20d74ce036be: Status 404 returned error can't find the container with id 1b83866927554d2bb21fc6bc1d43484e929aa51a09897e2b26ac20d74ce036be Oct 30 12:56:12.468022 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:12.467978 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-8dvmm" event={"ID":"da32ea91-80d7-48d5-9167-882b4736108e","Type":"ContainerStarted","Data":"1b83866927554d2bb21fc6bc1d43484e929aa51a09897e2b26ac20d74ce036be"} Oct 30 12:56:12.916878 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:12.916846 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg"] Oct 30 12:56:12.920493 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:12.920470 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:12.942133 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:12.942103 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fqrgx\"" Oct 30 12:56:12.942277 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:12.942150 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 12:56:12.942277 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:12.942195 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 12:56:12.942277 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:12.942217 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 12:56:12.944564 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:12.944530 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg"] Oct 30 12:56:12.997193 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:12.997157 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 12:56:13.070636 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.070594 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.070824 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.070706 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.070824 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.070747 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f4728d-6a18-46ee-aef6-96b02794270b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.071011 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.070979 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30f4728d-6a18-46ee-aef6-96b02794270b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.071116 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.071099 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.071186 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.071152 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f4728d-6a18-46ee-aef6-96b02794270b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.071186 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.071177 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjchd\" (UniqueName: \"kubernetes.io/projected/30f4728d-6a18-46ee-aef6-96b02794270b-kube-api-access-wjchd\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.171998 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.171898 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.172169 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.172012 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f4728d-6a18-46ee-aef6-96b02794270b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.172169 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.172043 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjchd\" (UniqueName: \"kubernetes.io/projected/30f4728d-6a18-46ee-aef6-96b02794270b-kube-api-access-wjchd\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.172169 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.172083 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.172169 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.172126 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.172169 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.172156 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f4728d-6a18-46ee-aef6-96b02794270b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.172426 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.172184 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30f4728d-6a18-46ee-aef6-96b02794270b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.172984 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.172956 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30f4728d-6a18-46ee-aef6-96b02794270b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.175313 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.175285 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f4728d-6a18-46ee-aef6-96b02794270b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.175415 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.175313 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.175481 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.175415 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.177247 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.177228 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.197383 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.197360 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f4728d-6a18-46ee-aef6-96b02794270b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.197733 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.197709 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjchd\" (UniqueName: \"kubernetes.io/projected/30f4728d-6a18-46ee-aef6-96b02794270b-kube-api-access-wjchd\") pod \"istiod-openshift-gateway-7876bfcb74-mpftg\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.235769 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.235740 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:13.475444 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.475369 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-qlkgt" event={"ID":"ec8777f0-af67-4906-8537-631e712dd6b6","Type":"ContainerStarted","Data":"b5f274c8abd43db59f114961c5d5734b27a240fa197dffc871f95cde89a65b55"} Oct 30 12:56:13.475870 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.475530 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-qlkgt" Oct 30 12:56:13.538479 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.538423 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-qlkgt" podStartSLOduration=3.095997018 podStartE2EDuration="5.538406214s" totalImagesPullingTime="2.442409196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:13.537474245 +0000 UTC m=+267.360702522" watchObservedRunningTime="2026-10-30 12:56:13.538406214 +0000 UTC m=+267.361634492" Oct 30 12:56:13.601021 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:13.600992 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg"] Oct 30 12:56:13.602240 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:56:13.602206 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f4728d_6a18_46ee_aef6_96b02794270b.slice/crio-cdc14cb8773154387d367ce70f710583b8e48727160dac908795a9936cd8fb2c WatchSource:0}: Error finding container cdc14cb8773154387d367ce70f710583b8e48727160dac908795a9936cd8fb2c: Status 404 returned error can't find the container with id cdc14cb8773154387d367ce70f710583b8e48727160dac908795a9936cd8fb2c Oct 30 12:56:14.482979 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:14.482777 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-8dvmm" event={"ID":"da32ea91-80d7-48d5-9167-882b4736108e","Type":"ContainerStarted","Data":"90a459705edadd4ab12a895c9e903892d002eaa77382cf3b165e490184b02175"} Oct 30 12:56:14.484327 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:14.484299 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" event={"ID":"30f4728d-6a18-46ee-aef6-96b02794270b","Type":"ContainerStarted","Data":"cdc14cb8773154387d367ce70f710583b8e48727160dac908795a9936cd8fb2c"} Oct 30 12:56:14.528733 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:14.528678 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d59dd5dc7-8dvmm" podStartSLOduration=1.040898441 podStartE2EDuration="3.528664804s" totalImagesPullingTime="2.487766363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:14.528330493 +0000 UTC m=+268.351558774" watchObservedRunningTime="2026-10-30 12:56:14.528664804 +0000 UTC m=+268.351893084" Oct 30 12:56:16.048258 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:16.048225 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:56:16.048558 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:16.048289 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:56:16.498756 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:16.498716 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" event={"ID":"30f4728d-6a18-46ee-aef6-96b02794270b","Type":"ContainerStarted","Data":"9cebde8dd065dae2261405bcf2246b178ec801ebe8ea27df93a766fdb0bdd21b"} Oct 30 12:56:16.498917 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:16.498821 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:16.559829 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:16.559779 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" podStartSLOduration=2.117068466 podStartE2EDuration="4.559766677s" totalImagesPullingTime="2.442698211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:16.558425675 +0000 UTC m=+270.381653953" watchObservedRunningTime="2026-10-30 12:56:16.559766677 +0000 UTC m=+270.382994954" Oct 30 12:56:17.505497 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:17.505467 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:56:24.486754 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:56:24.486722 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-qlkgt" Oct 30 12:56:29.700107 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:56:29.700070 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 12:56:59.700166 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:56:59.700090 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 12:57:10.030779 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.030748 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q"] Oct 30 12:57:10.034170 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.034149 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.065746 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.065721 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q"] Oct 30 12:57:10.171157 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.171121 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.171327 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.171182 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.171327 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.171281 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.171327 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.171309 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5qt2\" (UniqueName: \"kubernetes.io/projected/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-kube-api-access-r5qt2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.171426 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.171366 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.171426 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.171384 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.171426 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.171406 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.272388 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.272353 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.272388 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.272394 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.272604 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.272417 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5qt2\" (UniqueName: \"kubernetes.io/projected/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-kube-api-access-r5qt2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.272604 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.272452 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.272604 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.272474 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.272604 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.272500 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.272604 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.272547 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.273552 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.273517 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.274851 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.274820 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.274984 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.274962 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.274984 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.274976 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.275074 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.275004 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.314146 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.314086 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.314261 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.314164 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5qt2\" (UniqueName: \"kubernetes.io/projected/dfe5fa31-1804-404f-8b06-3e3c2dd8e42e-kube-api-access-r5qt2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrf4q\" (UID: \"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.345625 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.345600 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:10.677614 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.677584 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q"] Oct 30 12:57:10.678280 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:57:10.678253 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe5fa31_1804_404f_8b06_3e3c2dd8e42e.slice/crio-b1fe134f7ce225a2eac868f7a6eba6ca2463e7c8407c973cc7ea8b65c6e2d213 WatchSource:0}: Error finding container b1fe134f7ce225a2eac868f7a6eba6ca2463e7c8407c973cc7ea8b65c6e2d213: Status 404 returned error can't find the container with id b1fe134f7ce225a2eac868f7a6eba6ca2463e7c8407c973cc7ea8b65c6e2d213 Oct 30 12:57:10.680358 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.680329 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:57:10.680453 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.680383 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 12:57:10.735099 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:10.735070 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" event={"ID":"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e","Type":"ContainerStarted","Data":"b1fe134f7ce225a2eac868f7a6eba6ca2463e7c8407c973cc7ea8b65c6e2d213"} Oct 30 12:57:11.741719 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:11.741683 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" event={"ID":"dfe5fa31-1804-404f-8b06-3e3c2dd8e42e","Type":"ContainerStarted","Data":"e17b298506b42d86751a2ad9afc1deb7e4196eb8606e606a523c08a287535e71"} Oct 30 12:57:11.742338 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:11.741848 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:11.743551 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:11.743523 4503 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wrf4q container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:57:11.743667 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:11.743572 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" podUID="dfe5fa31-1804-404f-8b06-3e3c2dd8e42e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:57:11.821794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:11.821728 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" podStartSLOduration=2.8217145869999998 podStartE2EDuration="2.821714587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:11.808413157 +0000 UTC m=+325.631641438" watchObservedRunningTime="2026-10-30 12:57:11.821714587 +0000 UTC m=+325.644942865" Oct 30 12:57:12.747647 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:12.747618 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrf4q" Oct 30 12:57:12.852724 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:12.852691 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg"] Oct 30 12:57:12.852974 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:12.852951 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" podUID="30f4728d-6a18-46ee-aef6-96b02794270b" containerName="discovery" containerID="cri-o://9cebde8dd065dae2261405bcf2246b178ec801ebe8ea27df93a766fdb0bdd21b" gracePeriod=30 Oct 30 12:57:13.092215 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.092193 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:57:13.200514 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.200485 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-istio-csr-dns-cert\") pod \"30f4728d-6a18-46ee-aef6-96b02794270b\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " Oct 30 12:57:13.200695 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.200524 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f4728d-6a18-46ee-aef6-96b02794270b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30f4728d-6a18-46ee-aef6-96b02794270b-istio-csr-ca-configmap\") pod \"30f4728d-6a18-46ee-aef6-96b02794270b\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " Oct 30 12:57:13.200695 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.200567 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-istio-kubeconfig\") pod \"30f4728d-6a18-46ee-aef6-96b02794270b\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " Oct 30 12:57:13.200695 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.200591 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-cacerts\" (UniqueName: \"kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-cacerts\") pod \"30f4728d-6a18-46ee-aef6-96b02794270b\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " Oct 30 12:57:13.200695 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.200628 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f4728d-6a18-46ee-aef6-96b02794270b-istio-token\" (UniqueName: \"kubernetes.io/projected/30f4728d-6a18-46ee-aef6-96b02794270b-istio-token\") pod \"30f4728d-6a18-46ee-aef6-96b02794270b\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " Oct 30 12:57:13.200695 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.200677 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f4728d-6a18-46ee-aef6-96b02794270b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f4728d-6a18-46ee-aef6-96b02794270b-local-certs\") pod \"30f4728d-6a18-46ee-aef6-96b02794270b\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " Oct 30 12:57:13.200979 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.200757 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f4728d-6a18-46ee-aef6-96b02794270b-kube-api-access-wjchd\" (UniqueName: \"kubernetes.io/projected/30f4728d-6a18-46ee-aef6-96b02794270b-kube-api-access-wjchd\") pod \"30f4728d-6a18-46ee-aef6-96b02794270b\" (UID: \"30f4728d-6a18-46ee-aef6-96b02794270b\") " Oct 30 12:57:13.201048 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.200994 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f4728d-6a18-46ee-aef6-96b02794270b-istio-csr-ca-configmap" pod "30f4728d-6a18-46ee-aef6-96b02794270b" (UID: "30f4728d-6a18-46ee-aef6-96b02794270b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:57:13.203052 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.203021 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-istio-csr-dns-cert" pod "30f4728d-6a18-46ee-aef6-96b02794270b" (UID: "30f4728d-6a18-46ee-aef6-96b02794270b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:57:13.203197 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.203120 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f4728d-6a18-46ee-aef6-96b02794270b-local-certs" pod "30f4728d-6a18-46ee-aef6-96b02794270b" (UID: "30f4728d-6a18-46ee-aef6-96b02794270b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:57:13.203257 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.203237 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-istio-kubeconfig" pod "30f4728d-6a18-46ee-aef6-96b02794270b" (UID: "30f4728d-6a18-46ee-aef6-96b02794270b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:57:13.203334 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.203305 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-cacerts" pod "30f4728d-6a18-46ee-aef6-96b02794270b" (UID: "30f4728d-6a18-46ee-aef6-96b02794270b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:57:13.203334 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.203318 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f4728d-6a18-46ee-aef6-96b02794270b-istio-token" pod "30f4728d-6a18-46ee-aef6-96b02794270b" (UID: "30f4728d-6a18-46ee-aef6-96b02794270b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:57:13.203512 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.203396 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f4728d-6a18-46ee-aef6-96b02794270b-kube-api-access-wjchd" pod "30f4728d-6a18-46ee-aef6-96b02794270b" (UID: "30f4728d-6a18-46ee-aef6-96b02794270b"). InnerVolumeSpecName "kube-api-access-wjchd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:57:13.301858 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.301790 4503 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-istio-csr-dns-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:57:13.301858 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.301814 4503 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30f4728d-6a18-46ee-aef6-96b02794270b-istio-csr-ca-configmap\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:57:13.301858 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.301824 4503 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-istio-kubeconfig\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:57:13.301858 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.301833 4503 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30f4728d-6a18-46ee-aef6-96b02794270b-cacerts\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:57:13.301858 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.301841 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f4728d-6a18-46ee-aef6-96b02794270b-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:57:13.301858 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.301850 4503 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f4728d-6a18-46ee-aef6-96b02794270b-local-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:57:13.301858 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.301859 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjchd\" (UniqueName: \"kubernetes.io/projected/30f4728d-6a18-46ee-aef6-96b02794270b-kube-api-access-wjchd\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:57:13.752660 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.752630 4503 generic.go:356] "Generic (PLEG): container finished" podID="30f4728d-6a18-46ee-aef6-96b02794270b" containerID="9cebde8dd065dae2261405bcf2246b178ec801ebe8ea27df93a766fdb0bdd21b" exitCode=0 Oct 30 12:57:13.753094 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.752676 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" Oct 30 12:57:13.753094 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.752716 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" event={"ID":"30f4728d-6a18-46ee-aef6-96b02794270b","Type":"ContainerDied","Data":"9cebde8dd065dae2261405bcf2246b178ec801ebe8ea27df93a766fdb0bdd21b"} Oct 30 12:57:13.753094 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.752774 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg" event={"ID":"30f4728d-6a18-46ee-aef6-96b02794270b","Type":"ContainerDied","Data":"cdc14cb8773154387d367ce70f710583b8e48727160dac908795a9936cd8fb2c"} Oct 30 12:57:13.753094 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.752798 4503 scope.go:122] "RemoveContainer" containerID="9cebde8dd065dae2261405bcf2246b178ec801ebe8ea27df93a766fdb0bdd21b" Oct 30 12:57:13.762048 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.762030 4503 scope.go:122] "RemoveContainer" containerID="9cebde8dd065dae2261405bcf2246b178ec801ebe8ea27df93a766fdb0bdd21b" Oct 30 12:57:13.762337 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:57:13.762317 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cebde8dd065dae2261405bcf2246b178ec801ebe8ea27df93a766fdb0bdd21b\": container with ID starting with 9cebde8dd065dae2261405bcf2246b178ec801ebe8ea27df93a766fdb0bdd21b not found: ID does not exist" containerID="9cebde8dd065dae2261405bcf2246b178ec801ebe8ea27df93a766fdb0bdd21b" Oct 30 12:57:13.762405 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.762349 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cebde8dd065dae2261405bcf2246b178ec801ebe8ea27df93a766fdb0bdd21b"} err="failed to get container status \"9cebde8dd065dae2261405bcf2246b178ec801ebe8ea27df93a766fdb0bdd21b\": rpc error: code = NotFound desc = could not find container \"9cebde8dd065dae2261405bcf2246b178ec801ebe8ea27df93a766fdb0bdd21b\": container with ID starting with 9cebde8dd065dae2261405bcf2246b178ec801ebe8ea27df93a766fdb0bdd21b not found: ID does not exist" Oct 30 12:57:13.814853 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.814820 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg"] Oct 30 12:57:13.827537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:13.827504 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mpftg"] Oct 30 12:57:14.360324 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:57:14.360292 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f4728d-6a18-46ee-aef6-96b02794270b" path="/var/lib/kubelet/pods/30f4728d-6a18-46ee-aef6-96b02794270b/volumes" Oct 30 12:57:29.699738 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:57:29.699697 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 12:57:59.699321 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:57:59.699279 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 12:58:01.019145 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.019112 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn"] Oct 30 12:58:01.019626 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.019445 4503 cpu_manager.go:414] "containerMap: removing container" podUID="30f4728d-6a18-46ee-aef6-96b02794270b" containerName="discovery" Oct 30 12:58:01.019626 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.019457 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f4728d-6a18-46ee-aef6-96b02794270b" containerName="discovery" Oct 30 12:58:01.019626 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.019535 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f4728d-6a18-46ee-aef6-96b02794270b" containerName="discovery" Oct 30 12:58:01.022654 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.022637 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn" Oct 30 12:58:01.050923 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.050887 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 12:58:01.050923 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.050886 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 12:58:01.051122 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.050973 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-jslhr\"" Oct 30 12:58:01.052778 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.052748 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn"] Oct 30 12:58:01.092984 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.092932 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c4647c6-c54f-4a84-ae6f-adb482fcf727-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn\" (UID: \"2c4647c6-c54f-4a84-ae6f-adb482fcf727\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn" Oct 30 12:58:01.093147 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.092992 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c4647c6-c54f-4a84-ae6f-adb482fcf727-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn\" (UID: \"2c4647c6-c54f-4a84-ae6f-adb482fcf727\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn" Oct 30 12:58:01.093147 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.093065 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7852f\" (UniqueName: \"kubernetes.io/projected/2c4647c6-c54f-4a84-ae6f-adb482fcf727-kube-api-access-7852f\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn\" (UID: \"2c4647c6-c54f-4a84-ae6f-adb482fcf727\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn" Oct 30 12:58:01.115954 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.115917 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg"] Oct 30 12:58:01.119423 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.119407 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg" Oct 30 12:58:01.139801 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.139777 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-vskqt\"" Oct 30 12:58:01.141062 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.141040 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg"] Oct 30 12:58:01.193790 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.193750 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c4647c6-c54f-4a84-ae6f-adb482fcf727-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn\" (UID: \"2c4647c6-c54f-4a84-ae6f-adb482fcf727\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn" Oct 30 12:58:01.193790 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.193791 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c4647c6-c54f-4a84-ae6f-adb482fcf727-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn\" (UID: \"2c4647c6-c54f-4a84-ae6f-adb482fcf727\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn" Oct 30 12:58:01.194009 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.193822 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7852f\" (UniqueName: \"kubernetes.io/projected/2c4647c6-c54f-4a84-ae6f-adb482fcf727-kube-api-access-7852f\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn\" (UID: \"2c4647c6-c54f-4a84-ae6f-adb482fcf727\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn" Oct 30 12:58:01.194009 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.193847 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6kxf\" (UniqueName: \"kubernetes.io/projected/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-kube-api-access-j6kxf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg\" (UID: \"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg" Oct 30 12:58:01.194009 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.193914 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg\" (UID: \"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg" Oct 30 12:58:01.194009 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.193977 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg\" (UID: \"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg" Oct 30 12:58:01.194231 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.194209 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c4647c6-c54f-4a84-ae6f-adb482fcf727-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn\" (UID: \"2c4647c6-c54f-4a84-ae6f-adb482fcf727\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn" Oct 30 12:58:01.194277 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.194218 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c4647c6-c54f-4a84-ae6f-adb482fcf727-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn\" (UID: \"2c4647c6-c54f-4a84-ae6f-adb482fcf727\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn" Oct 30 12:58:01.207483 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.207457 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs"] Oct 30 12:58:01.211074 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.211053 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs" Oct 30 12:58:01.219679 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.219648 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7852f\" (UniqueName: \"kubernetes.io/projected/2c4647c6-c54f-4a84-ae6f-adb482fcf727-kube-api-access-7852f\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn\" (UID: \"2c4647c6-c54f-4a84-ae6f-adb482fcf727\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn" Oct 30 12:58:01.226583 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.226562 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-cbkmb\"" Oct 30 12:58:01.237747 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.237721 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs"] Oct 30 12:58:01.295290 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.295200 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb6d927d-300a-47c0-9100-a575ae8621d9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs\" (UID: \"bb6d927d-300a-47c0-9100-a575ae8621d9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs" Oct 30 12:58:01.295290 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.295248 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb6d927d-300a-47c0-9100-a575ae8621d9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs\" (UID: \"bb6d927d-300a-47c0-9100-a575ae8621d9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs" Oct 30 12:58:01.295290 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.295270 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88wlj\" (UniqueName: \"kubernetes.io/projected/bb6d927d-300a-47c0-9100-a575ae8621d9-kube-api-access-88wlj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs\" (UID: \"bb6d927d-300a-47c0-9100-a575ae8621d9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs" Oct 30 12:58:01.295590 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.295473 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg\" (UID: \"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg" Oct 30 12:58:01.295590 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.295525 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg\" (UID: \"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg" Oct 30 12:58:01.295691 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.295594 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6kxf\" (UniqueName: \"kubernetes.io/projected/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-kube-api-access-j6kxf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg\" (UID: \"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg" Oct 30 12:58:01.295838 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.295814 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg\" (UID: \"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg" Oct 30 12:58:01.295907 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.295848 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg\" (UID: \"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg" Oct 30 12:58:01.322657 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.322633 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7"] Oct 30 12:58:01.326218 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.326201 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7" Oct 30 12:58:01.327557 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.327535 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6kxf\" (UniqueName: \"kubernetes.io/projected/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-kube-api-access-j6kxf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg\" (UID: \"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg" Oct 30 12:58:01.338204 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.338187 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn" Oct 30 12:58:01.345592 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.345569 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-cpxk6\"" Oct 30 12:58:01.346381 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.346359 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7"] Oct 30 12:58:01.397264 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.396330 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d32ba6a4-3533-40f5-988b-fa7636f5d02d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7\" (UID: \"d32ba6a4-3533-40f5-988b-fa7636f5d02d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7" Oct 30 12:58:01.397264 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.396413 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d32ba6a4-3533-40f5-988b-fa7636f5d02d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7\" (UID: \"d32ba6a4-3533-40f5-988b-fa7636f5d02d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7" Oct 30 12:58:01.397264 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.396460 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb6d927d-300a-47c0-9100-a575ae8621d9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs\" (UID: \"bb6d927d-300a-47c0-9100-a575ae8621d9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs" Oct 30 12:58:01.397264 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.396494 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb6d927d-300a-47c0-9100-a575ae8621d9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs\" (UID: \"bb6d927d-300a-47c0-9100-a575ae8621d9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs" Oct 30 12:58:01.397264 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.396522 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88wlj\" (UniqueName: \"kubernetes.io/projected/bb6d927d-300a-47c0-9100-a575ae8621d9-kube-api-access-88wlj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs\" (UID: \"bb6d927d-300a-47c0-9100-a575ae8621d9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs" Oct 30 12:58:01.397264 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.396560 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5rw5\" (UniqueName: \"kubernetes.io/projected/d32ba6a4-3533-40f5-988b-fa7636f5d02d-kube-api-access-r5rw5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7\" (UID: \"d32ba6a4-3533-40f5-988b-fa7636f5d02d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7" Oct 30 12:58:01.397264 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.397077 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb6d927d-300a-47c0-9100-a575ae8621d9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs\" (UID: \"bb6d927d-300a-47c0-9100-a575ae8621d9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs" Oct 30 12:58:01.397264 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.397228 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb6d927d-300a-47c0-9100-a575ae8621d9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs\" (UID: \"bb6d927d-300a-47c0-9100-a575ae8621d9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs" Oct 30 12:58:01.422372 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.422340 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88wlj\" (UniqueName: \"kubernetes.io/projected/bb6d927d-300a-47c0-9100-a575ae8621d9-kube-api-access-88wlj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs\" (UID: \"bb6d927d-300a-47c0-9100-a575ae8621d9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs" Oct 30 12:58:01.435288 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.435260 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg" Oct 30 12:58:01.497108 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.497072 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d32ba6a4-3533-40f5-988b-fa7636f5d02d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7\" (UID: \"d32ba6a4-3533-40f5-988b-fa7636f5d02d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7" Oct 30 12:58:01.497253 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.497148 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5rw5\" (UniqueName: \"kubernetes.io/projected/d32ba6a4-3533-40f5-988b-fa7636f5d02d-kube-api-access-r5rw5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7\" (UID: \"d32ba6a4-3533-40f5-988b-fa7636f5d02d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7" Oct 30 12:58:01.497253 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.497246 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d32ba6a4-3533-40f5-988b-fa7636f5d02d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7\" (UID: \"d32ba6a4-3533-40f5-988b-fa7636f5d02d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7" Oct 30 12:58:01.497595 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.497568 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d32ba6a4-3533-40f5-988b-fa7636f5d02d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7\" (UID: \"d32ba6a4-3533-40f5-988b-fa7636f5d02d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7" Oct 30 12:58:01.497663 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.497602 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d32ba6a4-3533-40f5-988b-fa7636f5d02d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7\" (UID: \"d32ba6a4-3533-40f5-988b-fa7636f5d02d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7" Oct 30 12:58:01.523380 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.523354 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5rw5\" (UniqueName: \"kubernetes.io/projected/d32ba6a4-3533-40f5-988b-fa7636f5d02d-kube-api-access-r5rw5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7\" (UID: \"d32ba6a4-3533-40f5-988b-fa7636f5d02d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7" Oct 30 12:58:01.528924 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.528897 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs" Oct 30 12:58:01.639488 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.639430 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn"] Oct 30 12:58:01.650231 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.650208 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7" Oct 30 12:58:01.733835 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.733808 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg"] Oct 30 12:58:01.736584 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:58:01.736553 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6f5f1e_7465_4c84_ad47_3f82b7052e6b.slice/crio-9bec9b9d0526aaab9655932583d7a24236c65e20846a738533a22eaccbcd83ee WatchSource:0}: Error finding container 9bec9b9d0526aaab9655932583d7a24236c65e20846a738533a22eaccbcd83ee: Status 404 returned error can't find the container with id 9bec9b9d0526aaab9655932583d7a24236c65e20846a738533a22eaccbcd83ee Oct 30 12:58:01.832394 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.829203 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs"] Oct 30 12:58:01.832394 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:58:01.830048 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6d927d_300a_47c0_9100_a575ae8621d9.slice/crio-019c738e1560d438073da78450ad2dce12ed35c1cb46b60c5d3588934bbf3338 WatchSource:0}: Error finding container 019c738e1560d438073da78450ad2dce12ed35c1cb46b60c5d3588934bbf3338: Status 404 returned error can't find the container with id 019c738e1560d438073da78450ad2dce12ed35c1cb46b60c5d3588934bbf3338 Oct 30 12:58:01.968262 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.968232 4503 generic.go:356] "Generic (PLEG): container finished" podID="2c4647c6-c54f-4a84-ae6f-adb482fcf727" containerID="612f74948a3ff1377e647e5a96be1dafc72e3b9b9c3ec623ba678d5381b3e229" exitCode=0 Oct 30 12:58:01.968420 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.968319 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn" event={"ID":"2c4647c6-c54f-4a84-ae6f-adb482fcf727","Type":"ContainerDied","Data":"612f74948a3ff1377e647e5a96be1dafc72e3b9b9c3ec623ba678d5381b3e229"} Oct 30 12:58:01.968420 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.968376 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn" event={"ID":"2c4647c6-c54f-4a84-ae6f-adb482fcf727","Type":"ContainerStarted","Data":"f6758740ba91a1ccfaa4fad7168238dcbd544d3d8d7dad6cbe327c95b740abd1"} Oct 30 12:58:01.969575 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.969553 4503 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:58:01.969880 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.969862 4503 generic.go:356] "Generic (PLEG): container finished" podID="bb6d927d-300a-47c0-9100-a575ae8621d9" containerID="46a96105dfbc4ec887aad46db5d6bb0b27f8cf6c36c3933bc6f0d7a37e4a6055" exitCode=0 Oct 30 12:58:01.969990 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.969967 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs" event={"ID":"bb6d927d-300a-47c0-9100-a575ae8621d9","Type":"ContainerDied","Data":"46a96105dfbc4ec887aad46db5d6bb0b27f8cf6c36c3933bc6f0d7a37e4a6055"} Oct 30 12:58:01.970097 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.970011 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs" event={"ID":"bb6d927d-300a-47c0-9100-a575ae8621d9","Type":"ContainerStarted","Data":"019c738e1560d438073da78450ad2dce12ed35c1cb46b60c5d3588934bbf3338"} Oct 30 12:58:01.972237 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.972216 4503 generic.go:356] "Generic (PLEG): container finished" podID="0e6f5f1e-7465-4c84-ad47-3f82b7052e6b" containerID="f5e337bb559d12d6ff61184069f9afe959819b1f8fb7fbaf9cc0d704a3f77837" exitCode=0 Oct 30 12:58:01.972345 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.972242 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg" event={"ID":"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b","Type":"ContainerDied","Data":"f5e337bb559d12d6ff61184069f9afe959819b1f8fb7fbaf9cc0d704a3f77837"} Oct 30 12:58:01.972345 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.972292 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg" event={"ID":"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b","Type":"ContainerStarted","Data":"9bec9b9d0526aaab9655932583d7a24236c65e20846a738533a22eaccbcd83ee"} Oct 30 12:58:01.975158 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:01.975136 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7"] Oct 30 12:58:01.998770 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:58:01.998742 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32ba6a4_3533_40f5_988b_fa7636f5d02d.slice/crio-5a5046224fe929114ab8da193afd4a362c0861b253070c43265031302854094f WatchSource:0}: Error finding container 5a5046224fe929114ab8da193afd4a362c0861b253070c43265031302854094f: Status 404 returned error can't find the container with id 5a5046224fe929114ab8da193afd4a362c0861b253070c43265031302854094f Oct 30 12:58:02.981628 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:02.981546 4503 generic.go:356] "Generic (PLEG): container finished" podID="2c4647c6-c54f-4a84-ae6f-adb482fcf727" containerID="2e6ea1c746c516673760c07221d0b546b5ede70898aed787c00fd5413de13b97" exitCode=0 Oct 30 12:58:02.982085 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:02.981632 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn" event={"ID":"2c4647c6-c54f-4a84-ae6f-adb482fcf727","Type":"ContainerDied","Data":"2e6ea1c746c516673760c07221d0b546b5ede70898aed787c00fd5413de13b97"} Oct 30 12:58:02.983466 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:02.983433 4503 generic.go:356] "Generic (PLEG): container finished" podID="bb6d927d-300a-47c0-9100-a575ae8621d9" containerID="141d9edc9f092856b31e37896eaa87456144ea635f2707dc1a8d0241840c8679" exitCode=0 Oct 30 12:58:02.983588 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:02.983477 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs" event={"ID":"bb6d927d-300a-47c0-9100-a575ae8621d9","Type":"ContainerDied","Data":"141d9edc9f092856b31e37896eaa87456144ea635f2707dc1a8d0241840c8679"} Oct 30 12:58:02.985672 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:02.985650 4503 generic.go:356] "Generic (PLEG): container finished" podID="0e6f5f1e-7465-4c84-ad47-3f82b7052e6b" containerID="c91279dd127c547827d5a38b12974628b90c8b2e69e34fdfaab0fdc65163f1d0" exitCode=0 Oct 30 12:58:02.985777 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:02.985741 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg" event={"ID":"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b","Type":"ContainerDied","Data":"c91279dd127c547827d5a38b12974628b90c8b2e69e34fdfaab0fdc65163f1d0"} Oct 30 12:58:02.987301 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:02.987279 4503 generic.go:356] "Generic (PLEG): container finished" podID="d32ba6a4-3533-40f5-988b-fa7636f5d02d" containerID="4736ff2ab405e2a7d9e2ac629a3644e385f041cd17624a14cf12cd3eb192f3c7" exitCode=0 Oct 30 12:58:02.987404 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:02.987339 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7" event={"ID":"d32ba6a4-3533-40f5-988b-fa7636f5d02d","Type":"ContainerDied","Data":"4736ff2ab405e2a7d9e2ac629a3644e385f041cd17624a14cf12cd3eb192f3c7"} Oct 30 12:58:02.987404 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:02.987377 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7" event={"ID":"d32ba6a4-3533-40f5-988b-fa7636f5d02d","Type":"ContainerStarted","Data":"5a5046224fe929114ab8da193afd4a362c0861b253070c43265031302854094f"} Oct 30 12:58:03.994086 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:03.994048 4503 generic.go:356] "Generic (PLEG): container finished" podID="bb6d927d-300a-47c0-9100-a575ae8621d9" containerID="c9324ee9d316d83a0b5b96cff89ac9640c542e96ec3d51a1ffbff11ccbf7a906" exitCode=0 Oct 30 12:58:03.994531 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:03.994129 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs" event={"ID":"bb6d927d-300a-47c0-9100-a575ae8621d9","Type":"ContainerDied","Data":"c9324ee9d316d83a0b5b96cff89ac9640c542e96ec3d51a1ffbff11ccbf7a906"} Oct 30 12:58:03.996059 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:03.996028 4503 generic.go:356] "Generic (PLEG): container finished" podID="0e6f5f1e-7465-4c84-ad47-3f82b7052e6b" containerID="3d744e64ec456f9ba7c3f928dde766097be575bff400b685f8a6fef392f7c7f4" exitCode=0 Oct 30 12:58:03.996194 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:03.996107 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg" event={"ID":"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b","Type":"ContainerDied","Data":"3d744e64ec456f9ba7c3f928dde766097be575bff400b685f8a6fef392f7c7f4"} Oct 30 12:58:03.997697 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:03.997678 4503 generic.go:356] "Generic (PLEG): container finished" podID="d32ba6a4-3533-40f5-988b-fa7636f5d02d" containerID="0d0912349936796fa314d73738ab84c5f12f60704c6c2ca48c97a680a3fa5ad8" exitCode=0 Oct 30 12:58:03.997786 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:03.997754 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7" event={"ID":"d32ba6a4-3533-40f5-988b-fa7636f5d02d","Type":"ContainerDied","Data":"0d0912349936796fa314d73738ab84c5f12f60704c6c2ca48c97a680a3fa5ad8"} Oct 30 12:58:03.999768 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:03.999748 4503 generic.go:356] "Generic (PLEG): container finished" podID="2c4647c6-c54f-4a84-ae6f-adb482fcf727" containerID="1fea8a7c9e9997dd74ae5751d99fc1cafdbe8f6728bc328a077d77cc7fd6147b" exitCode=0 Oct 30 12:58:03.999854 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:03.999779 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn" event={"ID":"2c4647c6-c54f-4a84-ae6f-adb482fcf727","Type":"ContainerDied","Data":"1fea8a7c9e9997dd74ae5751d99fc1cafdbe8f6728bc328a077d77cc7fd6147b"} Oct 30 12:58:05.006789 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.006747 4503 generic.go:356] "Generic (PLEG): container finished" podID="d32ba6a4-3533-40f5-988b-fa7636f5d02d" containerID="43bdaeb0325bfb30f936f09e14e9a1c8df5ded657cb18436655688dd99480f27" exitCode=0 Oct 30 12:58:05.007226 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.006822 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7" event={"ID":"d32ba6a4-3533-40f5-988b-fa7636f5d02d","Type":"ContainerDied","Data":"43bdaeb0325bfb30f936f09e14e9a1c8df5ded657cb18436655688dd99480f27"} Oct 30 12:58:05.173423 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.173393 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn" Oct 30 12:58:05.176191 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.176168 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs" Oct 30 12:58:05.191276 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.191252 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg" Oct 30 12:58:05.333242 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.333165 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c4647c6-c54f-4a84-ae6f-adb482fcf727-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c4647c6-c54f-4a84-ae6f-adb482fcf727-bundle\") pod \"2c4647c6-c54f-4a84-ae6f-adb482fcf727\" (UID: \"2c4647c6-c54f-4a84-ae6f-adb482fcf727\") " Oct 30 12:58:05.333242 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.333221 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c4647c6-c54f-4a84-ae6f-adb482fcf727-kube-api-access-7852f\" (UniqueName: \"kubernetes.io/projected/2c4647c6-c54f-4a84-ae6f-adb482fcf727-kube-api-access-7852f\") pod \"2c4647c6-c54f-4a84-ae6f-adb482fcf727\" (UID: \"2c4647c6-c54f-4a84-ae6f-adb482fcf727\") " Oct 30 12:58:05.333420 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.333257 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6d927d-300a-47c0-9100-a575ae8621d9-util\" (UniqueName: \"kubernetes.io/empty-dir/bb6d927d-300a-47c0-9100-a575ae8621d9-util\") pod \"bb6d927d-300a-47c0-9100-a575ae8621d9\" (UID: \"bb6d927d-300a-47c0-9100-a575ae8621d9\") " Oct 30 12:58:05.333420 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.333277 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6d927d-300a-47c0-9100-a575ae8621d9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb6d927d-300a-47c0-9100-a575ae8621d9-bundle\") pod \"bb6d927d-300a-47c0-9100-a575ae8621d9\" (UID: \"bb6d927d-300a-47c0-9100-a575ae8621d9\") " Oct 30 12:58:05.333420 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.333333 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6d927d-300a-47c0-9100-a575ae8621d9-kube-api-access-88wlj\" (UniqueName: \"kubernetes.io/projected/bb6d927d-300a-47c0-9100-a575ae8621d9-kube-api-access-88wlj\") pod \"bb6d927d-300a-47c0-9100-a575ae8621d9\" (UID: \"bb6d927d-300a-47c0-9100-a575ae8621d9\") " Oct 30 12:58:05.333420 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.333361 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-util\" (UniqueName: \"kubernetes.io/empty-dir/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-util\") pod \"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b\" (UID: \"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b\") " Oct 30 12:58:05.333420 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.333415 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c4647c6-c54f-4a84-ae6f-adb482fcf727-util\" (UniqueName: \"kubernetes.io/empty-dir/2c4647c6-c54f-4a84-ae6f-adb482fcf727-util\") pod \"2c4647c6-c54f-4a84-ae6f-adb482fcf727\" (UID: \"2c4647c6-c54f-4a84-ae6f-adb482fcf727\") " Oct 30 12:58:05.333680 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.333449 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-kube-api-access-j6kxf\" (UniqueName: \"kubernetes.io/projected/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-kube-api-access-j6kxf\") pod \"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b\" (UID: \"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b\") " Oct 30 12:58:05.333680 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.333506 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-bundle\") pod \"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b\" (UID: \"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b\") " Oct 30 12:58:05.334053 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.333887 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4647c6-c54f-4a84-ae6f-adb482fcf727-bundle" pod "2c4647c6-c54f-4a84-ae6f-adb482fcf727" (UID: "2c4647c6-c54f-4a84-ae6f-adb482fcf727"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:58:05.334217 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.334134 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6d927d-300a-47c0-9100-a575ae8621d9-bundle" pod "bb6d927d-300a-47c0-9100-a575ae8621d9" (UID: "bb6d927d-300a-47c0-9100-a575ae8621d9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:58:05.334602 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.334575 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-bundle" pod "0e6f5f1e-7465-4c84-ad47-3f82b7052e6b" (UID: "0e6f5f1e-7465-4c84-ad47-3f82b7052e6b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:58:05.336098 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.336075 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4647c6-c54f-4a84-ae6f-adb482fcf727-kube-api-access-7852f" pod "2c4647c6-c54f-4a84-ae6f-adb482fcf727" (UID: "2c4647c6-c54f-4a84-ae6f-adb482fcf727"). InnerVolumeSpecName "kube-api-access-7852f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:58:05.336198 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.336108 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-kube-api-access-j6kxf" pod "0e6f5f1e-7465-4c84-ad47-3f82b7052e6b" (UID: "0e6f5f1e-7465-4c84-ad47-3f82b7052e6b"). InnerVolumeSpecName "kube-api-access-j6kxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:58:05.336528 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.336507 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6d927d-300a-47c0-9100-a575ae8621d9-kube-api-access-88wlj" pod "bb6d927d-300a-47c0-9100-a575ae8621d9" (UID: "bb6d927d-300a-47c0-9100-a575ae8621d9"). InnerVolumeSpecName "kube-api-access-88wlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:58:05.342484 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.342444 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6d927d-300a-47c0-9100-a575ae8621d9-util" pod "bb6d927d-300a-47c0-9100-a575ae8621d9" (UID: "bb6d927d-300a-47c0-9100-a575ae8621d9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:58:05.342848 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.342831 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-util" pod "0e6f5f1e-7465-4c84-ad47-3f82b7052e6b" (UID: "0e6f5f1e-7465-4c84-ad47-3f82b7052e6b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:58:05.343380 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.343348 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4647c6-c54f-4a84-ae6f-adb482fcf727-util" pod "2c4647c6-c54f-4a84-ae6f-adb482fcf727" (UID: "2c4647c6-c54f-4a84-ae6f-adb482fcf727"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:58:05.434297 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.434265 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7852f\" (UniqueName: \"kubernetes.io/projected/2c4647c6-c54f-4a84-ae6f-adb482fcf727-kube-api-access-7852f\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:58:05.434297 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.434293 4503 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb6d927d-300a-47c0-9100-a575ae8621d9-util\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:58:05.434297 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.434304 4503 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb6d927d-300a-47c0-9100-a575ae8621d9-bundle\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:58:05.434517 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.434312 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88wlj\" (UniqueName: \"kubernetes.io/projected/bb6d927d-300a-47c0-9100-a575ae8621d9-kube-api-access-88wlj\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:58:05.434517 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.434321 4503 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-util\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:58:05.434517 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.434330 4503 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c4647c6-c54f-4a84-ae6f-adb482fcf727-util\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:58:05.434517 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.434340 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6kxf\" (UniqueName: \"kubernetes.io/projected/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-kube-api-access-j6kxf\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:58:05.434517 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.434349 4503 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e6f5f1e-7465-4c84-ad47-3f82b7052e6b-bundle\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:58:05.434517 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:05.434357 4503 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c4647c6-c54f-4a84-ae6f-adb482fcf727-bundle\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:58:06.014441 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:06.014410 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn" Oct 30 12:58:06.014952 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:06.014410 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w69qn" event={"ID":"2c4647c6-c54f-4a84-ae6f-adb482fcf727","Type":"ContainerDied","Data":"f6758740ba91a1ccfaa4fad7168238dcbd544d3d8d7dad6cbe327c95b740abd1"} Oct 30 12:58:06.014952 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:06.014530 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6758740ba91a1ccfaa4fad7168238dcbd544d3d8d7dad6cbe327c95b740abd1" Oct 30 12:58:06.016397 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:06.016377 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs" Oct 30 12:58:06.016496 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:06.016370 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6mwfs" event={"ID":"bb6d927d-300a-47c0-9100-a575ae8621d9","Type":"ContainerDied","Data":"019c738e1560d438073da78450ad2dce12ed35c1cb46b60c5d3588934bbf3338"} Oct 30 12:58:06.016559 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:06.016509 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="019c738e1560d438073da78450ad2dce12ed35c1cb46b60c5d3588934bbf3338" Oct 30 12:58:06.018494 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:06.018475 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg" Oct 30 12:58:06.018583 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:06.018506 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4vffg" event={"ID":"0e6f5f1e-7465-4c84-ad47-3f82b7052e6b","Type":"ContainerDied","Data":"9bec9b9d0526aaab9655932583d7a24236c65e20846a738533a22eaccbcd83ee"} Oct 30 12:58:06.018583 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:06.018545 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bec9b9d0526aaab9655932583d7a24236c65e20846a738533a22eaccbcd83ee" Oct 30 12:58:06.146123 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:06.146098 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7" Oct 30 12:58:06.241202 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:06.241175 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32ba6a4-3533-40f5-988b-fa7636f5d02d-kube-api-access-r5rw5\" (UniqueName: \"kubernetes.io/projected/d32ba6a4-3533-40f5-988b-fa7636f5d02d-kube-api-access-r5rw5\") pod \"d32ba6a4-3533-40f5-988b-fa7636f5d02d\" (UID: \"d32ba6a4-3533-40f5-988b-fa7636f5d02d\") " Oct 30 12:58:06.241394 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:06.241273 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32ba6a4-3533-40f5-988b-fa7636f5d02d-util\" (UniqueName: \"kubernetes.io/empty-dir/d32ba6a4-3533-40f5-988b-fa7636f5d02d-util\") pod \"d32ba6a4-3533-40f5-988b-fa7636f5d02d\" (UID: \"d32ba6a4-3533-40f5-988b-fa7636f5d02d\") " Oct 30 12:58:06.241394 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:06.241360 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32ba6a4-3533-40f5-988b-fa7636f5d02d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d32ba6a4-3533-40f5-988b-fa7636f5d02d-bundle\") pod \"d32ba6a4-3533-40f5-988b-fa7636f5d02d\" (UID: \"d32ba6a4-3533-40f5-988b-fa7636f5d02d\") " Oct 30 12:58:06.241813 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:06.241784 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32ba6a4-3533-40f5-988b-fa7636f5d02d-bundle" pod "d32ba6a4-3533-40f5-988b-fa7636f5d02d" (UID: "d32ba6a4-3533-40f5-988b-fa7636f5d02d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:58:06.243390 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:06.243359 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32ba6a4-3533-40f5-988b-fa7636f5d02d-kube-api-access-r5rw5" pod "d32ba6a4-3533-40f5-988b-fa7636f5d02d" (UID: "d32ba6a4-3533-40f5-988b-fa7636f5d02d"). InnerVolumeSpecName "kube-api-access-r5rw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:58:06.246166 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:06.246134 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32ba6a4-3533-40f5-988b-fa7636f5d02d-util" pod "d32ba6a4-3533-40f5-988b-fa7636f5d02d" (UID: "d32ba6a4-3533-40f5-988b-fa7636f5d02d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:58:06.342795 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:06.342715 4503 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d32ba6a4-3533-40f5-988b-fa7636f5d02d-util\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:58:06.342795 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:06.342743 4503 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d32ba6a4-3533-40f5-988b-fa7636f5d02d-bundle\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:58:06.342795 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:06.342752 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5rw5\" (UniqueName: \"kubernetes.io/projected/d32ba6a4-3533-40f5-988b-fa7636f5d02d-kube-api-access-r5rw5\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:58:07.025035 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:07.025006 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7" Oct 30 12:58:07.025404 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:07.025010 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lzz7" event={"ID":"d32ba6a4-3533-40f5-988b-fa7636f5d02d","Type":"ContainerDied","Data":"5a5046224fe929114ab8da193afd4a362c0861b253070c43265031302854094f"} Oct 30 12:58:07.025404 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:07.025117 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a5046224fe929114ab8da193afd4a362c0861b253070c43265031302854094f" Oct 30 12:58:12.352158 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352124 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm"] Oct 30 12:58:12.352547 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352452 4503 cpu_manager.go:414] "containerMap: removing container" podUID="d32ba6a4-3533-40f5-988b-fa7636f5d02d" containerName="util" Oct 30 12:58:12.352547 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352463 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32ba6a4-3533-40f5-988b-fa7636f5d02d" containerName="util" Oct 30 12:58:12.352547 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352475 4503 cpu_manager.go:414] "containerMap: removing container" podUID="2c4647c6-c54f-4a84-ae6f-adb482fcf727" containerName="extract" Oct 30 12:58:12.352547 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352480 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c4647c6-c54f-4a84-ae6f-adb482fcf727" containerName="extract" Oct 30 12:58:12.352547 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352490 4503 cpu_manager.go:414] "containerMap: removing container" podUID="0e6f5f1e-7465-4c84-ad47-3f82b7052e6b" containerName="extract" Oct 30 12:58:12.352547 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352495 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e6f5f1e-7465-4c84-ad47-3f82b7052e6b" containerName="extract" Oct 30 12:58:12.352547 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352502 4503 cpu_manager.go:414] "containerMap: removing container" podUID="2c4647c6-c54f-4a84-ae6f-adb482fcf727" containerName="pull" Oct 30 12:58:12.352547 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352506 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c4647c6-c54f-4a84-ae6f-adb482fcf727" containerName="pull" Oct 30 12:58:12.352547 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352515 4503 cpu_manager.go:414] "containerMap: removing container" podUID="bb6d927d-300a-47c0-9100-a575ae8621d9" containerName="extract" Oct 30 12:58:12.352547 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352519 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6d927d-300a-47c0-9100-a575ae8621d9" containerName="extract" Oct 30 12:58:12.352547 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352532 4503 cpu_manager.go:414] "containerMap: removing container" podUID="d32ba6a4-3533-40f5-988b-fa7636f5d02d" containerName="pull" Oct 30 12:58:12.352547 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352536 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32ba6a4-3533-40f5-988b-fa7636f5d02d" containerName="pull" Oct 30 12:58:12.352547 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352545 4503 cpu_manager.go:414] "containerMap: removing container" podUID="2c4647c6-c54f-4a84-ae6f-adb482fcf727" containerName="util" Oct 30 12:58:12.352547 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352549 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c4647c6-c54f-4a84-ae6f-adb482fcf727" containerName="util" Oct 30 12:58:12.352547 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352555 4503 cpu_manager.go:414] "containerMap: removing container" podUID="0e6f5f1e-7465-4c84-ad47-3f82b7052e6b" containerName="util" Oct 30 12:58:12.352977 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352560 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e6f5f1e-7465-4c84-ad47-3f82b7052e6b" containerName="util" Oct 30 12:58:12.352977 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352566 4503 cpu_manager.go:414] "containerMap: removing container" podUID="d32ba6a4-3533-40f5-988b-fa7636f5d02d" containerName="extract" Oct 30 12:58:12.352977 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352570 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32ba6a4-3533-40f5-988b-fa7636f5d02d" containerName="extract" Oct 30 12:58:12.352977 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352580 4503 cpu_manager.go:414] "containerMap: removing container" podUID="bb6d927d-300a-47c0-9100-a575ae8621d9" containerName="util" Oct 30 12:58:12.352977 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352584 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6d927d-300a-47c0-9100-a575ae8621d9" containerName="util" Oct 30 12:58:12.352977 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352592 4503 cpu_manager.go:414] "containerMap: removing container" podUID="0e6f5f1e-7465-4c84-ad47-3f82b7052e6b" containerName="pull" Oct 30 12:58:12.352977 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352596 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e6f5f1e-7465-4c84-ad47-3f82b7052e6b" containerName="pull" Oct 30 12:58:12.352977 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352602 4503 cpu_manager.go:414] "containerMap: removing container" podUID="bb6d927d-300a-47c0-9100-a575ae8621d9" containerName="pull" Oct 30 12:58:12.352977 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352606 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6d927d-300a-47c0-9100-a575ae8621d9" containerName="pull" Oct 30 12:58:12.352977 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352649 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32ba6a4-3533-40f5-988b-fa7636f5d02d" containerName="extract" Oct 30 12:58:12.352977 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352658 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c4647c6-c54f-4a84-ae6f-adb482fcf727" containerName="extract" Oct 30 12:58:12.352977 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352665 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e6f5f1e-7465-4c84-ad47-3f82b7052e6b" containerName="extract" Oct 30 12:58:12.352977 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.352671 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6d927d-300a-47c0-9100-a575ae8621d9" containerName="extract" Oct 30 12:58:12.355974 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.355928 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm" Oct 30 12:58:12.377017 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.376964 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 12:58:12.388229 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.388207 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-vfg7c\"" Oct 30 12:58:12.389927 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.389907 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm"] Oct 30 12:58:12.482269 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.482233 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 12:58:12.493639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.493612 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w6mv\" (UniqueName: \"kubernetes.io/projected/de1f5d1e-3bd9-48cf-b307-e9768c988f37-kube-api-access-7w6mv\") pod \"limitador-operator-controller-manager-856dd584c5-c9scm\" (UID: \"de1f5d1e-3bd9-48cf-b307-e9768c988f37\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm" Oct 30 12:58:12.595076 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.595035 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w6mv\" (UniqueName: \"kubernetes.io/projected/de1f5d1e-3bd9-48cf-b307-e9768c988f37-kube-api-access-7w6mv\") pod \"limitador-operator-controller-manager-856dd584c5-c9scm\" (UID: \"de1f5d1e-3bd9-48cf-b307-e9768c988f37\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm" Oct 30 12:58:12.623471 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.623386 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w6mv\" (UniqueName: \"kubernetes.io/projected/de1f5d1e-3bd9-48cf-b307-e9768c988f37-kube-api-access-7w6mv\") pod \"limitador-operator-controller-manager-856dd584c5-c9scm\" (UID: \"de1f5d1e-3bd9-48cf-b307-e9768c988f37\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm" Oct 30 12:58:12.668470 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.668428 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm" Oct 30 12:58:12.967434 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:12.967409 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm"] Oct 30 12:58:12.968870 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:58:12.968839 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde1f5d1e_3bd9_48cf_b307_e9768c988f37.slice/crio-e7f2395b20185cafab686ebccabeed2688bb9f793b91c8bbd8fac7be213852f1 WatchSource:0}: Error finding container e7f2395b20185cafab686ebccabeed2688bb9f793b91c8bbd8fac7be213852f1: Status 404 returned error can't find the container with id e7f2395b20185cafab686ebccabeed2688bb9f793b91c8bbd8fac7be213852f1 Oct 30 12:58:13.053644 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:13.053610 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm" event={"ID":"de1f5d1e-3bd9-48cf-b307-e9768c988f37","Type":"ContainerStarted","Data":"e7f2395b20185cafab686ebccabeed2688bb9f793b91c8bbd8fac7be213852f1"} Oct 30 12:58:16.072080 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:16.072033 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm" event={"ID":"de1f5d1e-3bd9-48cf-b307-e9768c988f37","Type":"ContainerStarted","Data":"6fd7c8219e18ce09671436ce5939f108b247849e7228c59d9d310194d8891e41"} Oct 30 12:58:16.072487 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:16.072112 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm" Oct 30 12:58:16.117888 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:16.117839 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm" podStartSLOduration=1.568915406 podStartE2EDuration="4.117825008s" totalImagesPullingTime="2.548909602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:16.116246346 +0000 UTC m=+389.939474614" watchObservedRunningTime="2026-10-30 12:58:16.117825008 +0000 UTC m=+389.941053286" Oct 30 12:58:16.478009 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:16.477980 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6"] Oct 30 12:58:16.481228 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:16.481209 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" Oct 30 12:58:16.503627 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:16.503594 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-t2snd\"" Oct 30 12:58:16.515956 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:16.515913 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6"] Oct 30 12:58:16.522868 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:16.522840 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qts6k\" (UniqueName: \"kubernetes.io/projected/0eb718b1-6abe-4401-b568-780ba3636ae8-kube-api-access-qts6k\") pod \"kuadrant-operator-controller-manager-6b6fb56846-6fhc6\" (UID: \"0eb718b1-6abe-4401-b568-780ba3636ae8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" Oct 30 12:58:16.523052 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:16.522896 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0eb718b1-6abe-4401-b568-780ba3636ae8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-6fhc6\" (UID: \"0eb718b1-6abe-4401-b568-780ba3636ae8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" Oct 30 12:58:16.623987 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:16.623950 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qts6k\" (UniqueName: \"kubernetes.io/projected/0eb718b1-6abe-4401-b568-780ba3636ae8-kube-api-access-qts6k\") pod \"kuadrant-operator-controller-manager-6b6fb56846-6fhc6\" (UID: \"0eb718b1-6abe-4401-b568-780ba3636ae8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" Oct 30 12:58:16.624173 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:16.624029 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0eb718b1-6abe-4401-b568-780ba3636ae8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-6fhc6\" (UID: \"0eb718b1-6abe-4401-b568-780ba3636ae8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" Oct 30 12:58:16.624457 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:16.624436 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0eb718b1-6abe-4401-b568-780ba3636ae8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-6fhc6\" (UID: \"0eb718b1-6abe-4401-b568-780ba3636ae8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" Oct 30 12:58:16.648805 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:16.648775 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qts6k\" (UniqueName: \"kubernetes.io/projected/0eb718b1-6abe-4401-b568-780ba3636ae8-kube-api-access-qts6k\") pod \"kuadrant-operator-controller-manager-6b6fb56846-6fhc6\" (UID: \"0eb718b1-6abe-4401-b568-780ba3636ae8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" Oct 30 12:58:16.794073 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:16.793980 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" Oct 30 12:58:17.133735 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:17.133701 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6"] Oct 30 12:58:18.082826 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:18.082782 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" event={"ID":"0eb718b1-6abe-4401-b568-780ba3636ae8","Type":"ContainerStarted","Data":"1530240e775431450c0fd864e77a5a2c081cc330f554654b167bec9ee8023f8c"} Oct 30 12:58:21.101387 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:21.101350 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" event={"ID":"0eb718b1-6abe-4401-b568-780ba3636ae8","Type":"ContainerStarted","Data":"ec33e3289a68c7daf9f33da3bd8296abf443e6601d8901704869cb52addbacae"} Oct 30 12:58:21.101854 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:21.101487 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" Oct 30 12:58:21.174757 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:21.174689 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" podStartSLOduration=1.297268469 podStartE2EDuration="5.174668602s" totalImagesPullingTime="3.877400133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:21.160296066 +0000 UTC m=+394.983524343" watchObservedRunningTime="2026-10-30 12:58:21.174668602 +0000 UTC m=+394.997896881" Oct 30 12:58:27.078827 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:27.078792 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm" Oct 30 12:58:29.699590 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:58:29.699512 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 12:58:32.109051 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:32.109019 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" Oct 30 12:58:35.897795 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:35.897763 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6"] Oct 30 12:58:35.898255 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:35.897966 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" podUID="0eb718b1-6abe-4401-b568-780ba3636ae8" containerName="manager" containerID="cri-o://ec33e3289a68c7daf9f33da3bd8296abf443e6601d8901704869cb52addbacae" gracePeriod=2 Oct 30 12:58:35.938722 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:35.938691 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6"] Oct 30 12:58:35.971397 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:58:35.971351 4503 status_manager.go:1068] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0eb718b1-6abe-4401-b568-780ba3636ae8\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-6b6fb56846-6fhc6\": pods \"kuadrant-operator-controller-manager-6b6fb56846-6fhc6\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" Oct 30 12:58:35.974170 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:35.974141 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm"] Oct 30 12:58:35.974432 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:35.974383 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm" podUID="de1f5d1e-3bd9-48cf-b307-e9768c988f37" containerName="manager" containerID="cri-o://6fd7c8219e18ce09671436ce5939f108b247849e7228c59d9d310194d8891e41" gracePeriod=2 Oct 30 12:58:35.987963 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:58:35.987918 4503 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-6fhc6\" is forbidden: User \"system:node:ip-10-0-130-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-200.ec2.internal' and this object" podUID="0eb718b1-6abe-4401-b568-780ba3636ae8" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" Oct 30 12:58:36.015280 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.015248 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm"] Oct 30 12:58:36.017208 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.017180 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-czj25"] Oct 30 12:58:36.017555 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.017538 4503 cpu_manager.go:414] "containerMap: removing container" podUID="de1f5d1e-3bd9-48cf-b307-e9768c988f37" containerName="manager" Oct 30 12:58:36.017631 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.017557 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de1f5d1e-3bd9-48cf-b307-e9768c988f37" containerName="manager" Oct 30 12:58:36.017631 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.017596 4503 cpu_manager.go:414] "containerMap: removing container" podUID="0eb718b1-6abe-4401-b568-780ba3636ae8" containerName="manager" Oct 30 12:58:36.017631 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.017605 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb718b1-6abe-4401-b568-780ba3636ae8" containerName="manager" Oct 30 12:58:36.017766 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.017709 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="de1f5d1e-3bd9-48cf-b307-e9768c988f37" containerName="manager" Oct 30 12:58:36.017766 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.017727 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb718b1-6abe-4401-b568-780ba3636ae8" containerName="manager" Oct 30 12:58:36.020740 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.020719 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-czj25" Oct 30 12:58:36.025573 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:58:36.025544 4503 status_manager.go:1068] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"de1f5d1e-3bd9-48cf-b307-e9768c988f37\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-856dd584c5-c9scm\": pods \"limitador-operator-controller-manager-856dd584c5-c9scm\" not found" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm" Oct 30 12:58:36.039346 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:58:36.039177 4503 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-6fhc6\" is forbidden: User \"system:node:ip-10-0-130-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-200.ec2.internal' and this object" podUID="0eb718b1-6abe-4401-b568-780ba3636ae8" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" Oct 30 12:58:36.043830 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.043774 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-czj25"] Oct 30 12:58:36.084836 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.084810 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tvbm\" (UniqueName: \"kubernetes.io/projected/9b175e7a-d7b0-4425-9879-2d256d2f0324-kube-api-access-7tvbm\") pod \"limitador-operator-controller-manager-856dd584c5-czj25\" (UID: \"9b175e7a-d7b0-4425-9879-2d256d2f0324\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-czj25" Oct 30 12:58:36.102742 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:58:36.102700 4503 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-c9scm\" is forbidden: User \"system:node:ip-10-0-130-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-200.ec2.internal' and this object" podUID="de1f5d1e-3bd9-48cf-b307-e9768c988f37" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm" Oct 30 12:58:36.176471 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.176440 4503 generic.go:356] "Generic (PLEG): container finished" podID="0eb718b1-6abe-4401-b568-780ba3636ae8" containerID="ec33e3289a68c7daf9f33da3bd8296abf443e6601d8901704869cb52addbacae" exitCode=0 Oct 30 12:58:36.178471 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.178437 4503 generic.go:356] "Generic (PLEG): container finished" podID="de1f5d1e-3bd9-48cf-b307-e9768c988f37" containerID="6fd7c8219e18ce09671436ce5939f108b247849e7228c59d9d310194d8891e41" exitCode=0 Oct 30 12:58:36.185529 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.185507 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tvbm\" (UniqueName: \"kubernetes.io/projected/9b175e7a-d7b0-4425-9879-2d256d2f0324-kube-api-access-7tvbm\") pod \"limitador-operator-controller-manager-856dd584c5-czj25\" (UID: \"9b175e7a-d7b0-4425-9879-2d256d2f0324\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-czj25" Oct 30 12:58:36.213825 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.213805 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tvbm\" (UniqueName: \"kubernetes.io/projected/9b175e7a-d7b0-4425-9879-2d256d2f0324-kube-api-access-7tvbm\") pod \"limitador-operator-controller-manager-856dd584c5-czj25\" (UID: \"9b175e7a-d7b0-4425-9879-2d256d2f0324\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-czj25" Oct 30 12:58:36.227660 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.227642 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" Oct 30 12:58:36.230522 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.230506 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm" Oct 30 12:58:36.241262 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:58:36.241238 4503 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-6fhc6\" is forbidden: User \"system:node:ip-10-0-130-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-200.ec2.internal' and this object" podUID="0eb718b1-6abe-4401-b568-780ba3636ae8" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" Oct 30 12:58:36.254773 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:58:36.254749 4503 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-c9scm\" is forbidden: User \"system:node:ip-10-0-130-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-200.ec2.internal' and this object" podUID="de1f5d1e-3bd9-48cf-b307-e9768c988f37" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm" Oct 30 12:58:36.268174 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:58:36.268144 4503 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-c9scm\" is forbidden: User \"system:node:ip-10-0-130-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-200.ec2.internal' and this object" podUID="de1f5d1e-3bd9-48cf-b307-e9768c988f37" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm" Oct 30 12:58:36.282031 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:58:36.282007 4503 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-6fhc6\" is forbidden: User \"system:node:ip-10-0-130-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-200.ec2.internal' and this object" podUID="0eb718b1-6abe-4401-b568-780ba3636ae8" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" Oct 30 12:58:36.286272 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.286244 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de1f5d1e-3bd9-48cf-b307-e9768c988f37-kube-api-access-7w6mv\" (UniqueName: \"kubernetes.io/projected/de1f5d1e-3bd9-48cf-b307-e9768c988f37-kube-api-access-7w6mv\") pod \"de1f5d1e-3bd9-48cf-b307-e9768c988f37\" (UID: \"de1f5d1e-3bd9-48cf-b307-e9768c988f37\") " Oct 30 12:58:36.286338 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.286285 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb718b1-6abe-4401-b568-780ba3636ae8-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0eb718b1-6abe-4401-b568-780ba3636ae8-extensions-socket-volume\") pod \"0eb718b1-6abe-4401-b568-780ba3636ae8\" (UID: \"0eb718b1-6abe-4401-b568-780ba3636ae8\") " Oct 30 12:58:36.286390 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.286348 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb718b1-6abe-4401-b568-780ba3636ae8-kube-api-access-qts6k\" (UniqueName: \"kubernetes.io/projected/0eb718b1-6abe-4401-b568-780ba3636ae8-kube-api-access-qts6k\") pod \"0eb718b1-6abe-4401-b568-780ba3636ae8\" (UID: \"0eb718b1-6abe-4401-b568-780ba3636ae8\") " Oct 30 12:58:36.286804 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.286774 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb718b1-6abe-4401-b568-780ba3636ae8-extensions-socket-volume" pod "0eb718b1-6abe-4401-b568-780ba3636ae8" (UID: "0eb718b1-6abe-4401-b568-780ba3636ae8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:58:36.288571 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.288551 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb718b1-6abe-4401-b568-780ba3636ae8-kube-api-access-qts6k" pod "0eb718b1-6abe-4401-b568-780ba3636ae8" (UID: "0eb718b1-6abe-4401-b568-780ba3636ae8"). InnerVolumeSpecName "kube-api-access-qts6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:58:36.288639 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.288593 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1f5d1e-3bd9-48cf-b307-e9768c988f37-kube-api-access-7w6mv" pod "de1f5d1e-3bd9-48cf-b307-e9768c988f37" (UID: "de1f5d1e-3bd9-48cf-b307-e9768c988f37"). InnerVolumeSpecName "kube-api-access-7w6mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:58:36.355109 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.355073 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-czj25" Oct 30 12:58:36.364183 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.364149 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb718b1-6abe-4401-b568-780ba3636ae8" path="/var/lib/kubelet/pods/0eb718b1-6abe-4401-b568-780ba3636ae8/volumes" Oct 30 12:58:36.364735 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.364701 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de1f5d1e-3bd9-48cf-b307-e9768c988f37" path="/var/lib/kubelet/pods/de1f5d1e-3bd9-48cf-b307-e9768c988f37/volumes" Oct 30 12:58:36.371731 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:58:36.371697 4503 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-6fhc6\" is forbidden: User \"system:node:ip-10-0-130-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-200.ec2.internal' and this object" podUID="0eb718b1-6abe-4401-b568-780ba3636ae8" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" Oct 30 12:58:36.385406 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:58:36.385372 4503 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-c9scm\" is forbidden: User \"system:node:ip-10-0-130-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-200.ec2.internal' and this object" podUID="de1f5d1e-3bd9-48cf-b307-e9768c988f37" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm" Oct 30 12:58:36.387974 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.387933 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qts6k\" (UniqueName: \"kubernetes.io/projected/0eb718b1-6abe-4401-b568-780ba3636ae8-kube-api-access-qts6k\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:58:36.388077 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.387981 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w6mv\" (UniqueName: \"kubernetes.io/projected/de1f5d1e-3bd9-48cf-b307-e9768c988f37-kube-api-access-7w6mv\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:58:36.388077 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.387996 4503 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0eb718b1-6abe-4401-b568-780ba3636ae8-extensions-socket-volume\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:58:36.658533 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:36.658506 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-czj25"] Oct 30 12:58:36.659362 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:58:36.659333 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b175e7a_d7b0_4425_9879_2d256d2f0324.slice/crio-7b3d8aa8345af6b2d03baced63e7a2178aff31dc6d1a9ef813e7eb2193750bb1 WatchSource:0}: Error finding container 7b3d8aa8345af6b2d03baced63e7a2178aff31dc6d1a9ef813e7eb2193750bb1: Status 404 returned error can't find the container with id 7b3d8aa8345af6b2d03baced63e7a2178aff31dc6d1a9ef813e7eb2193750bb1 Oct 30 12:58:37.184841 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:37.184768 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-6fhc6" Oct 30 12:58:37.185319 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:37.184768 4503 scope.go:122] "RemoveContainer" containerID="ec33e3289a68c7daf9f33da3bd8296abf443e6601d8901704869cb52addbacae" Oct 30 12:58:37.186250 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:37.186231 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-c9scm" Oct 30 12:58:37.188090 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:37.188063 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-czj25" event={"ID":"9b175e7a-d7b0-4425-9879-2d256d2f0324","Type":"ContainerStarted","Data":"7b9e988b1e016010d1ebaca21121b66c3748412527c4b5f64e48635910ae399b"} Oct 30 12:58:37.188213 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:37.188120 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-czj25" event={"ID":"9b175e7a-d7b0-4425-9879-2d256d2f0324","Type":"ContainerStarted","Data":"7b3d8aa8345af6b2d03baced63e7a2178aff31dc6d1a9ef813e7eb2193750bb1"} Oct 30 12:58:37.188213 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:37.188144 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-czj25" Oct 30 12:58:37.193850 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:37.193833 4503 scope.go:122] "RemoveContainer" containerID="6fd7c8219e18ce09671436ce5939f108b247849e7228c59d9d310194d8891e41" Oct 30 12:58:37.244818 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:37.244771 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-czj25" podStartSLOduration=2.244756773 podStartE2EDuration="2.244756773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:37.243262922 +0000 UTC m=+411.066491199" watchObservedRunningTime="2026-10-30 12:58:37.244756773 +0000 UTC m=+411.067985049" Oct 30 12:58:37.836883 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:37.836843 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt"] Oct 30 12:58:37.842043 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:37.842020 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt" Oct 30 12:58:37.863656 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:37.863629 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt"] Oct 30 12:58:37.874173 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:37.874142 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-t2snd\"" Oct 30 12:58:37.902240 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:37.902210 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ecba625-9f77-49d0-9f9f-056fc89a6271-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4g8bt\" (UID: \"2ecba625-9f77-49d0-9f9f-056fc89a6271\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt" Oct 30 12:58:37.902240 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:37.902246 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf4mm\" (UniqueName: \"kubernetes.io/projected/2ecba625-9f77-49d0-9f9f-056fc89a6271-kube-api-access-hf4mm\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4g8bt\" (UID: \"2ecba625-9f77-49d0-9f9f-056fc89a6271\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt" Oct 30 12:58:38.003671 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:38.003630 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ecba625-9f77-49d0-9f9f-056fc89a6271-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4g8bt\" (UID: \"2ecba625-9f77-49d0-9f9f-056fc89a6271\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt" Oct 30 12:58:38.003671 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:38.003671 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf4mm\" (UniqueName: \"kubernetes.io/projected/2ecba625-9f77-49d0-9f9f-056fc89a6271-kube-api-access-hf4mm\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4g8bt\" (UID: \"2ecba625-9f77-49d0-9f9f-056fc89a6271\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt" Oct 30 12:58:38.004081 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:38.004058 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ecba625-9f77-49d0-9f9f-056fc89a6271-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4g8bt\" (UID: \"2ecba625-9f77-49d0-9f9f-056fc89a6271\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt" Oct 30 12:58:38.041992 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:38.041959 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf4mm\" (UniqueName: \"kubernetes.io/projected/2ecba625-9f77-49d0-9f9f-056fc89a6271-kube-api-access-hf4mm\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4g8bt\" (UID: \"2ecba625-9f77-49d0-9f9f-056fc89a6271\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt" Oct 30 12:58:38.154918 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:38.154890 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt" Oct 30 12:58:38.489996 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:38.489970 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt"] Oct 30 12:58:38.490646 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:58:38.490618 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ecba625_9f77_49d0_9f9f_056fc89a6271.slice/crio-4a51fa1dc1c43a863b8578aeeef51822297d4aa4c5ae9f51725e473b250bee69 WatchSource:0}: Error finding container 4a51fa1dc1c43a863b8578aeeef51822297d4aa4c5ae9f51725e473b250bee69: Status 404 returned error can't find the container with id 4a51fa1dc1c43a863b8578aeeef51822297d4aa4c5ae9f51725e473b250bee69 Oct 30 12:58:39.203589 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:39.203550 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt" event={"ID":"2ecba625-9f77-49d0-9f9f-056fc89a6271","Type":"ContainerStarted","Data":"f63eafd348c7a23caf5a9da5806c7b7834167f56f08d80174afd1d01eca9fd0d"} Oct 30 12:58:39.203776 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:39.203610 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt" event={"ID":"2ecba625-9f77-49d0-9f9f-056fc89a6271","Type":"ContainerStarted","Data":"4a51fa1dc1c43a863b8578aeeef51822297d4aa4c5ae9f51725e473b250bee69"} Oct 30 12:58:39.203776 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:39.203650 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt" Oct 30 12:58:39.280161 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:39.280107 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt" podStartSLOduration=2.280092901 podStartE2EDuration="2.280092901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:39.26484328 +0000 UTC m=+413.088071556" watchObservedRunningTime="2026-10-30 12:58:39.280092901 +0000 UTC m=+413.103321179" Oct 30 12:58:48.200056 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:48.200023 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-czj25" Oct 30 12:58:50.210767 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:58:50.210736 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt" Oct 30 12:58:59.699958 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:58:59.699911 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 12:59:10.856155 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:10.856122 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xdtbb"] Oct 30 12:59:10.859742 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:10.859721 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xdtbb" Oct 30 12:59:10.879842 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:10.879821 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xdtbb"] Oct 30 12:59:10.890111 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:10.890086 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-chvlr\"" Oct 30 12:59:10.911086 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:10.911059 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 12:59:10.990417 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:10.990372 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xdtbb"] Oct 30 12:59:10.990697 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:10.990677 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d-config-file\") pod \"limitador-limitador-74dfcfb894-xdtbb\" (UID: \"5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xdtbb" Oct 30 12:59:10.990819 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:10.990804 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jvr5\" (UniqueName: \"kubernetes.io/projected/5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d-kube-api-access-5jvr5\") pod \"limitador-limitador-74dfcfb894-xdtbb\" (UID: \"5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xdtbb" Oct 30 12:59:11.091653 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:11.091609 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jvr5\" (UniqueName: \"kubernetes.io/projected/5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d-kube-api-access-5jvr5\") pod \"limitador-limitador-74dfcfb894-xdtbb\" (UID: \"5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xdtbb" Oct 30 12:59:11.091811 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:11.091670 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d-config-file\") pod \"limitador-limitador-74dfcfb894-xdtbb\" (UID: \"5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xdtbb" Oct 30 12:59:11.092286 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:11.092268 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d-config-file\") pod \"limitador-limitador-74dfcfb894-xdtbb\" (UID: \"5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xdtbb" Oct 30 12:59:11.117883 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:11.117819 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jvr5\" (UniqueName: \"kubernetes.io/projected/5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d-kube-api-access-5jvr5\") pod \"limitador-limitador-74dfcfb894-xdtbb\" (UID: \"5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xdtbb" Oct 30 12:59:11.172133 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:11.172106 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xdtbb" Oct 30 12:59:11.469357 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:11.469320 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xdtbb"] Oct 30 12:59:11.469701 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:59:11.469675 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9e91cb_f107_4e6d_8d3e_ea859f0aa67d.slice/crio-a469c868493e9c04dc6ca2b97d4ccce42f0de47f835524dd097a95113500cdb2 WatchSource:0}: Error finding container a469c868493e9c04dc6ca2b97d4ccce42f0de47f835524dd097a95113500cdb2: Status 404 returned error can't find the container with id a469c868493e9c04dc6ca2b97d4ccce42f0de47f835524dd097a95113500cdb2 Oct 30 12:59:12.363990 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:12.363934 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xdtbb" event={"ID":"5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d","Type":"ContainerStarted","Data":"a469c868493e9c04dc6ca2b97d4ccce42f0de47f835524dd097a95113500cdb2"} Oct 30 12:59:13.370194 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:13.370159 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xdtbb" event={"ID":"5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d","Type":"ContainerStarted","Data":"b34a3d5607861a6d10e8eda3bc150437efda6cae667e94f630420f6097f3ff04"} Oct 30 12:59:13.370649 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:13.370255 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-xdtbb" Oct 30 12:59:13.413455 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:13.413407 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-xdtbb" podStartSLOduration=2.271818132 podStartE2EDuration="3.413393779s" totalImagesPullingTime="1.141575647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:13.412434274 +0000 UTC m=+447.235662552" watchObservedRunningTime="2026-10-30 12:59:13.413393779 +0000 UTC m=+447.236622055" Oct 30 12:59:24.376271 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:24.376243 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-xdtbb" Oct 30 12:59:25.312542 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:25.312508 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xdtbb"] Oct 30 12:59:25.312732 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:25.312705 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-xdtbb" podUID="5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d" containerName="limitador" containerID="cri-o://b34a3d5607861a6d10e8eda3bc150437efda6cae667e94f630420f6097f3ff04" gracePeriod=30 Oct 30 12:59:25.856537 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:25.856511 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xdtbb" Oct 30 12:59:25.896794 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:25.896771 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d-config-file\" (UniqueName: \"kubernetes.io/configmap/5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d-config-file\") pod \"5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d\" (UID: \"5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d\") " Oct 30 12:59:25.896927 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:25.896802 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d-kube-api-access-5jvr5\" (UniqueName: \"kubernetes.io/projected/5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d-kube-api-access-5jvr5\") pod \"5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d\" (UID: \"5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d\") " Oct 30 12:59:25.897136 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:25.897111 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d-config-file" pod "5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d" (UID: "5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:59:25.898867 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:25.898848 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d-kube-api-access-5jvr5" pod "5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d" (UID: "5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d"). InnerVolumeSpecName "kube-api-access-5jvr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:59:25.998003 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:25.997974 4503 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d-config-file\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:59:25.998003 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:25.998001 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jvr5\" (UniqueName: \"kubernetes.io/projected/5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d-kube-api-access-5jvr5\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 12:59:26.431140 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:26.431104 4503 generic.go:356] "Generic (PLEG): container finished" podID="5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d" containerID="b34a3d5607861a6d10e8eda3bc150437efda6cae667e94f630420f6097f3ff04" exitCode=0 Oct 30 12:59:26.431313 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:26.431146 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xdtbb" Oct 30 12:59:26.431313 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:26.431163 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xdtbb" event={"ID":"5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d","Type":"ContainerDied","Data":"b34a3d5607861a6d10e8eda3bc150437efda6cae667e94f630420f6097f3ff04"} Oct 30 12:59:26.431313 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:26.431207 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xdtbb" event={"ID":"5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d","Type":"ContainerDied","Data":"a469c868493e9c04dc6ca2b97d4ccce42f0de47f835524dd097a95113500cdb2"} Oct 30 12:59:26.431313 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:26.431227 4503 scope.go:122] "RemoveContainer" containerID="b34a3d5607861a6d10e8eda3bc150437efda6cae667e94f630420f6097f3ff04" Oct 30 12:59:26.439684 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:26.439666 4503 scope.go:122] "RemoveContainer" containerID="b34a3d5607861a6d10e8eda3bc150437efda6cae667e94f630420f6097f3ff04" Oct 30 12:59:26.439989 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:59:26.439966 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34a3d5607861a6d10e8eda3bc150437efda6cae667e94f630420f6097f3ff04\": container with ID starting with b34a3d5607861a6d10e8eda3bc150437efda6cae667e94f630420f6097f3ff04 not found: ID does not exist" containerID="b34a3d5607861a6d10e8eda3bc150437efda6cae667e94f630420f6097f3ff04" Oct 30 12:59:26.440083 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:26.440004 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34a3d5607861a6d10e8eda3bc150437efda6cae667e94f630420f6097f3ff04"} err="failed to get container status \"b34a3d5607861a6d10e8eda3bc150437efda6cae667e94f630420f6097f3ff04\": rpc error: code = NotFound desc = could not find container \"b34a3d5607861a6d10e8eda3bc150437efda6cae667e94f630420f6097f3ff04\": container with ID starting with b34a3d5607861a6d10e8eda3bc150437efda6cae667e94f630420f6097f3ff04 not found: ID does not exist" Oct 30 12:59:26.484954 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:26.484911 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xdtbb"] Oct 30 12:59:26.490961 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:26.490917 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xdtbb"] Oct 30 12:59:27.753426 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:27.753387 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qfpgx"] Oct 30 12:59:27.753998 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:27.753961 4503 cpu_manager.go:414] "containerMap: removing container" podUID="5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d" containerName="limitador" Oct 30 12:59:27.753998 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:27.753988 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d" containerName="limitador" Oct 30 12:59:27.754150 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:27.754099 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d" containerName="limitador" Oct 30 12:59:27.758467 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:27.758445 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qfpgx" Oct 30 12:59:27.778553 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:27.778528 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qfpgx"] Oct 30 12:59:27.788250 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:27.788223 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:59:27.788384 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:27.788352 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 12:59:27.788384 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:27.788364 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:59:27.788500 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:27.788476 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-msf2k\"" Oct 30 12:59:27.812663 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:27.812637 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bf42da6b-55a9-41e6-8881-da4580c09406-data\") pod \"postgres-7cdbf6dc44-qfpgx\" (UID: \"bf42da6b-55a9-41e6-8881-da4580c09406\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qfpgx" Oct 30 12:59:27.812790 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:27.812695 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtwdn\" (UniqueName: \"kubernetes.io/projected/bf42da6b-55a9-41e6-8881-da4580c09406-kube-api-access-dtwdn\") pod \"postgres-7cdbf6dc44-qfpgx\" (UID: \"bf42da6b-55a9-41e6-8881-da4580c09406\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qfpgx" Oct 30 12:59:27.913346 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:27.913308 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtwdn\" (UniqueName: \"kubernetes.io/projected/bf42da6b-55a9-41e6-8881-da4580c09406-kube-api-access-dtwdn\") pod \"postgres-7cdbf6dc44-qfpgx\" (UID: \"bf42da6b-55a9-41e6-8881-da4580c09406\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qfpgx" Oct 30 12:59:27.913520 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:27.913390 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bf42da6b-55a9-41e6-8881-da4580c09406-data\") pod \"postgres-7cdbf6dc44-qfpgx\" (UID: \"bf42da6b-55a9-41e6-8881-da4580c09406\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qfpgx" Oct 30 12:59:27.913810 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:27.913790 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bf42da6b-55a9-41e6-8881-da4580c09406-data\") pod \"postgres-7cdbf6dc44-qfpgx\" (UID: \"bf42da6b-55a9-41e6-8881-da4580c09406\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qfpgx" Oct 30 12:59:27.938330 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:27.938300 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtwdn\" (UniqueName: \"kubernetes.io/projected/bf42da6b-55a9-41e6-8881-da4580c09406-kube-api-access-dtwdn\") pod \"postgres-7cdbf6dc44-qfpgx\" (UID: \"bf42da6b-55a9-41e6-8881-da4580c09406\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qfpgx" Oct 30 12:59:28.071150 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:28.071063 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qfpgx" Oct 30 12:59:28.359590 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:28.359501 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d" path="/var/lib/kubelet/pods/5d9e91cb-f107-4e6d-8d3e-ea859f0aa67d/volumes" Oct 30 12:59:28.367895 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:59:28.367862 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf42da6b_55a9_41e6_8881_da4580c09406.slice/crio-da284ca23849176181186e7f1a29090e8eeea3cc0fd13a12ee83c95fdf52eba0 WatchSource:0}: Error finding container da284ca23849176181186e7f1a29090e8eeea3cc0fd13a12ee83c95fdf52eba0: Status 404 returned error can't find the container with id da284ca23849176181186e7f1a29090e8eeea3cc0fd13a12ee83c95fdf52eba0 Oct 30 12:59:28.368212 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:28.368189 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qfpgx"] Oct 30 12:59:28.447653 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:28.447618 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qfpgx" event={"ID":"bf42da6b-55a9-41e6-8881-da4580c09406","Type":"ContainerStarted","Data":"da284ca23849176181186e7f1a29090e8eeea3cc0fd13a12ee83c95fdf52eba0"} Oct 30 12:59:29.703507 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:59:29.703468 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 12:59:33.476410 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:33.476329 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qfpgx" event={"ID":"bf42da6b-55a9-41e6-8881-da4580c09406","Type":"ContainerStarted","Data":"85ee75568b12eccc92f263384e1867e093484b223a3ce90fec8a042077937c32"} Oct 30 12:59:33.476833 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:33.476520 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qfpgx" Oct 30 12:59:33.519108 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:33.519058 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qfpgx" podStartSLOduration=1.719548655 podStartE2EDuration="6.519042479s" totalImagesPullingTime="4.799493824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:33.51766633 +0000 UTC m=+467.340894609" watchObservedRunningTime="2026-10-30 12:59:33.519042479 +0000 UTC m=+467.342270756" Oct 30 12:59:39.510494 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:39.510465 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qfpgx" Oct 30 12:59:50.659963 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:50.659910 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-b6wzm"] Oct 30 12:59:50.682030 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:50.682003 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-b6wzm" Oct 30 12:59:50.696464 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:50.696409 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-b6wzm"] Oct 30 12:59:50.705591 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:50.705558 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 12:59:50.705591 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:50.705568 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 12:59:50.705791 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:50.705568 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 12:59:50.705871 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:50.705579 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-28684\"" Oct 30 12:59:50.804519 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:50.804485 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28420b22-5e86-41ae-b267-bf5688321d66-metrics-certs\") pod \"maas-controller-577f596567-b6wzm\" (UID: \"28420b22-5e86-41ae-b267-bf5688321d66\") " pod="opendatahub/maas-controller-577f596567-b6wzm" Oct 30 12:59:50.804687 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:50.804527 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28420b22-5e86-41ae-b267-bf5688321d66-webhook-cert\") pod \"maas-controller-577f596567-b6wzm\" (UID: \"28420b22-5e86-41ae-b267-bf5688321d66\") " pod="opendatahub/maas-controller-577f596567-b6wzm" Oct 30 12:59:50.804687 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:50.804621 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzpw\" (UniqueName: \"kubernetes.io/projected/28420b22-5e86-41ae-b267-bf5688321d66-kube-api-access-6pzpw\") pod \"maas-controller-577f596567-b6wzm\" (UID: \"28420b22-5e86-41ae-b267-bf5688321d66\") " pod="opendatahub/maas-controller-577f596567-b6wzm" Oct 30 12:59:50.905717 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:50.905681 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28420b22-5e86-41ae-b267-bf5688321d66-metrics-certs\") pod \"maas-controller-577f596567-b6wzm\" (UID: \"28420b22-5e86-41ae-b267-bf5688321d66\") " pod="opendatahub/maas-controller-577f596567-b6wzm" Oct 30 12:59:50.905717 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:50.905722 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28420b22-5e86-41ae-b267-bf5688321d66-webhook-cert\") pod \"maas-controller-577f596567-b6wzm\" (UID: \"28420b22-5e86-41ae-b267-bf5688321d66\") " pod="opendatahub/maas-controller-577f596567-b6wzm" Oct 30 12:59:50.905908 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:50.905748 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzpw\" (UniqueName: \"kubernetes.io/projected/28420b22-5e86-41ae-b267-bf5688321d66-kube-api-access-6pzpw\") pod \"maas-controller-577f596567-b6wzm\" (UID: \"28420b22-5e86-41ae-b267-bf5688321d66\") " pod="opendatahub/maas-controller-577f596567-b6wzm" Oct 30 12:59:50.908278 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:50.908252 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28420b22-5e86-41ae-b267-bf5688321d66-webhook-cert\") pod \"maas-controller-577f596567-b6wzm\" (UID: \"28420b22-5e86-41ae-b267-bf5688321d66\") " pod="opendatahub/maas-controller-577f596567-b6wzm" Oct 30 12:59:50.908362 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:50.908306 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28420b22-5e86-41ae-b267-bf5688321d66-metrics-certs\") pod \"maas-controller-577f596567-b6wzm\" (UID: \"28420b22-5e86-41ae-b267-bf5688321d66\") " pod="opendatahub/maas-controller-577f596567-b6wzm" Oct 30 12:59:50.930618 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:50.930555 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzpw\" (UniqueName: \"kubernetes.io/projected/28420b22-5e86-41ae-b267-bf5688321d66-kube-api-access-6pzpw\") pod \"maas-controller-577f596567-b6wzm\" (UID: \"28420b22-5e86-41ae-b267-bf5688321d66\") " pod="opendatahub/maas-controller-577f596567-b6wzm" Oct 30 12:59:50.999155 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:50.999125 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-b6wzm" Oct 30 12:59:51.074903 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:51.074876 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bd48f45d7-tscg6"] Oct 30 12:59:51.077728 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:51.077708 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd48f45d7-tscg6" Oct 30 12:59:51.101044 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:51.101003 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bd48f45d7-tscg6"] Oct 30 12:59:51.208514 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:51.208438 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnrt5\" (UniqueName: \"kubernetes.io/projected/30d64a87-af35-4a50-8779-fb9db0b61910-kube-api-access-bnrt5\") pod \"maas-controller-bd48f45d7-tscg6\" (UID: \"30d64a87-af35-4a50-8779-fb9db0b61910\") " pod="opendatahub/maas-controller-bd48f45d7-tscg6" Oct 30 12:59:51.208514 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:51.208490 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d64a87-af35-4a50-8779-fb9db0b61910-metrics-certs\") pod \"maas-controller-bd48f45d7-tscg6\" (UID: \"30d64a87-af35-4a50-8779-fb9db0b61910\") " pod="opendatahub/maas-controller-bd48f45d7-tscg6" Oct 30 12:59:51.208731 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:51.208637 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30d64a87-af35-4a50-8779-fb9db0b61910-webhook-cert\") pod \"maas-controller-bd48f45d7-tscg6\" (UID: \"30d64a87-af35-4a50-8779-fb9db0b61910\") " pod="opendatahub/maas-controller-bd48f45d7-tscg6" Oct 30 12:59:51.295737 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:51.295709 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-b6wzm"] Oct 30 12:59:51.296131 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:59:51.296108 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28420b22_5e86_41ae_b267_bf5688321d66.slice/crio-325f5be0f38363be7d516ec7e7ac6ef49a4a5a2a3655080d878fb91317a9167d WatchSource:0}: Error finding container 325f5be0f38363be7d516ec7e7ac6ef49a4a5a2a3655080d878fb91317a9167d: Status 404 returned error can't find the container with id 325f5be0f38363be7d516ec7e7ac6ef49a4a5a2a3655080d878fb91317a9167d Oct 30 12:59:51.309339 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:51.309318 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnrt5\" (UniqueName: \"kubernetes.io/projected/30d64a87-af35-4a50-8779-fb9db0b61910-kube-api-access-bnrt5\") pod \"maas-controller-bd48f45d7-tscg6\" (UID: \"30d64a87-af35-4a50-8779-fb9db0b61910\") " pod="opendatahub/maas-controller-bd48f45d7-tscg6" Oct 30 12:59:51.309427 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:51.309368 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d64a87-af35-4a50-8779-fb9db0b61910-metrics-certs\") pod \"maas-controller-bd48f45d7-tscg6\" (UID: \"30d64a87-af35-4a50-8779-fb9db0b61910\") " pod="opendatahub/maas-controller-bd48f45d7-tscg6" Oct 30 12:59:51.309427 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:51.309408 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30d64a87-af35-4a50-8779-fb9db0b61910-webhook-cert\") pod \"maas-controller-bd48f45d7-tscg6\" (UID: \"30d64a87-af35-4a50-8779-fb9db0b61910\") " pod="opendatahub/maas-controller-bd48f45d7-tscg6" Oct 30 12:59:51.311762 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:51.311741 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d64a87-af35-4a50-8779-fb9db0b61910-metrics-certs\") pod \"maas-controller-bd48f45d7-tscg6\" (UID: \"30d64a87-af35-4a50-8779-fb9db0b61910\") " pod="opendatahub/maas-controller-bd48f45d7-tscg6" Oct 30 12:59:51.311863 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:51.311780 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30d64a87-af35-4a50-8779-fb9db0b61910-webhook-cert\") pod \"maas-controller-bd48f45d7-tscg6\" (UID: \"30d64a87-af35-4a50-8779-fb9db0b61910\") " pod="opendatahub/maas-controller-bd48f45d7-tscg6" Oct 30 12:59:51.333736 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:51.333710 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnrt5\" (UniqueName: \"kubernetes.io/projected/30d64a87-af35-4a50-8779-fb9db0b61910-kube-api-access-bnrt5\") pod \"maas-controller-bd48f45d7-tscg6\" (UID: \"30d64a87-af35-4a50-8779-fb9db0b61910\") " pod="opendatahub/maas-controller-bd48f45d7-tscg6" Oct 30 12:59:51.389334 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:51.389310 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd48f45d7-tscg6" Oct 30 12:59:51.569744 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:51.569655 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-b6wzm" event={"ID":"28420b22-5e86-41ae-b267-bf5688321d66","Type":"ContainerStarted","Data":"325f5be0f38363be7d516ec7e7ac6ef49a4a5a2a3655080d878fb91317a9167d"} Oct 30 12:59:51.686028 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:51.685997 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bd48f45d7-tscg6"] Oct 30 12:59:51.686972 ip-10-0-130-200 kubenswrapper[4503]: W1030 12:59:51.686923 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d64a87_af35_4a50_8779_fb9db0b61910.slice/crio-f2b42a9eaf1940700f62440b260137b53a883c8a8eb9324b46c85f818ce60a6b WatchSource:0}: Error finding container f2b42a9eaf1940700f62440b260137b53a883c8a8eb9324b46c85f818ce60a6b: Status 404 returned error can't find the container with id f2b42a9eaf1940700f62440b260137b53a883c8a8eb9324b46c85f818ce60a6b Oct 30 12:59:52.579725 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:52.579688 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd48f45d7-tscg6" event={"ID":"30d64a87-af35-4a50-8779-fb9db0b61910","Type":"ContainerStarted","Data":"f2b42a9eaf1940700f62440b260137b53a883c8a8eb9324b46c85f818ce60a6b"} Oct 30 12:59:54.591741 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:54.591689 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-b6wzm" event={"ID":"28420b22-5e86-41ae-b267-bf5688321d66","Type":"ContainerStarted","Data":"977e6288a8f0b81c6ef15f113b6b30bbe8e61a6d564c97e6d8e405a829bcebc0"} Oct 30 12:59:54.592177 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:54.591766 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-b6wzm" Oct 30 12:59:54.593304 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:54.593276 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd48f45d7-tscg6" event={"ID":"30d64a87-af35-4a50-8779-fb9db0b61910","Type":"ContainerStarted","Data":"9885d39b921a2389046f33bcdd8cb83ded4c3ff2bae7dba0e2c86aaefe30e2ba"} Oct 30 12:59:54.593436 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:54.593404 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bd48f45d7-tscg6" Oct 30 12:59:54.635192 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:54.635086 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-b6wzm" podStartSLOduration=1.91991983 podStartE2EDuration="4.635070669s" totalImagesPullingTime="2.715150839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:54.634622493 +0000 UTC m=+488.457850771" watchObservedRunningTime="2026-10-30 12:59:54.635070669 +0000 UTC m=+488.458298943" Oct 30 12:59:54.678215 ip-10-0-130-200 kubenswrapper[4503]: I1030 12:59:54.678155 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-bd48f45d7-tscg6" podStartSLOduration=0.981005462 podStartE2EDuration="3.678134678s" totalImagesPullingTime="2.697129216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:54.677293883 +0000 UTC m=+488.500522160" watchObservedRunningTime="2026-10-30 12:59:54.678134678 +0000 UTC m=+488.501362958" Oct 30 12:59:59.699827 ip-10-0-130-200 kubenswrapper[4503]: E1030 12:59:59.699746 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:00:05.601031 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:05.600999 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bd48f45d7-tscg6" Oct 30 13:00:05.601494 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:05.601068 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-b6wzm" Oct 30 13:00:05.723270 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:05.723233 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-b6wzm"] Oct 30 13:00:05.723530 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:05.723491 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-b6wzm" podUID="28420b22-5e86-41ae-b267-bf5688321d66" containerName="manager" containerID="cri-o://977e6288a8f0b81c6ef15f113b6b30bbe8e61a6d564c97e6d8e405a829bcebc0" gracePeriod=30 Oct 30 13:00:05.962199 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:05.962175 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-b6wzm" Oct 30 13:00:06.149327 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:06.149291 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28420b22-5e86-41ae-b267-bf5688321d66-kube-api-access-6pzpw\" (UniqueName: \"kubernetes.io/projected/28420b22-5e86-41ae-b267-bf5688321d66-kube-api-access-6pzpw\") pod \"28420b22-5e86-41ae-b267-bf5688321d66\" (UID: \"28420b22-5e86-41ae-b267-bf5688321d66\") " Oct 30 13:00:06.149509 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:06.149357 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28420b22-5e86-41ae-b267-bf5688321d66-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28420b22-5e86-41ae-b267-bf5688321d66-metrics-certs\") pod \"28420b22-5e86-41ae-b267-bf5688321d66\" (UID: \"28420b22-5e86-41ae-b267-bf5688321d66\") " Oct 30 13:00:06.149509 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:06.149429 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28420b22-5e86-41ae-b267-bf5688321d66-webhook-cert\" (UniqueName: \"kubernetes.io/secret/28420b22-5e86-41ae-b267-bf5688321d66-webhook-cert\") pod \"28420b22-5e86-41ae-b267-bf5688321d66\" (UID: \"28420b22-5e86-41ae-b267-bf5688321d66\") " Oct 30 13:00:06.151587 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:06.151558 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28420b22-5e86-41ae-b267-bf5688321d66-metrics-certs" pod "28420b22-5e86-41ae-b267-bf5688321d66" (UID: "28420b22-5e86-41ae-b267-bf5688321d66"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:00:06.151697 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:06.151585 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28420b22-5e86-41ae-b267-bf5688321d66-webhook-cert" pod "28420b22-5e86-41ae-b267-bf5688321d66" (UID: "28420b22-5e86-41ae-b267-bf5688321d66"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:00:06.151737 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:06.151696 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28420b22-5e86-41ae-b267-bf5688321d66-kube-api-access-6pzpw" pod "28420b22-5e86-41ae-b267-bf5688321d66" (UID: "28420b22-5e86-41ae-b267-bf5688321d66"). InnerVolumeSpecName "kube-api-access-6pzpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:06.250821 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:06.250786 4503 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28420b22-5e86-41ae-b267-bf5688321d66-webhook-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:00:06.250821 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:06.250816 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pzpw\" (UniqueName: \"kubernetes.io/projected/28420b22-5e86-41ae-b267-bf5688321d66-kube-api-access-6pzpw\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:00:06.250821 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:06.250826 4503 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28420b22-5e86-41ae-b267-bf5688321d66-metrics-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:00:06.652814 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:06.652774 4503 generic.go:356] "Generic (PLEG): container finished" podID="28420b22-5e86-41ae-b267-bf5688321d66" containerID="977e6288a8f0b81c6ef15f113b6b30bbe8e61a6d564c97e6d8e405a829bcebc0" exitCode=0 Oct 30 13:00:06.653288 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:06.652811 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-b6wzm" event={"ID":"28420b22-5e86-41ae-b267-bf5688321d66","Type":"ContainerDied","Data":"977e6288a8f0b81c6ef15f113b6b30bbe8e61a6d564c97e6d8e405a829bcebc0"} Oct 30 13:00:06.653288 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:06.652850 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-b6wzm" Oct 30 13:00:06.653288 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:06.652880 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-b6wzm" event={"ID":"28420b22-5e86-41ae-b267-bf5688321d66","Type":"ContainerDied","Data":"325f5be0f38363be7d516ec7e7ac6ef49a4a5a2a3655080d878fb91317a9167d"} Oct 30 13:00:06.653288 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:06.652911 4503 scope.go:122] "RemoveContainer" containerID="977e6288a8f0b81c6ef15f113b6b30bbe8e61a6d564c97e6d8e405a829bcebc0" Oct 30 13:00:06.661608 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:06.661587 4503 scope.go:122] "RemoveContainer" containerID="977e6288a8f0b81c6ef15f113b6b30bbe8e61a6d564c97e6d8e405a829bcebc0" Oct 30 13:00:06.661898 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:00:06.661875 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"977e6288a8f0b81c6ef15f113b6b30bbe8e61a6d564c97e6d8e405a829bcebc0\": container with ID starting with 977e6288a8f0b81c6ef15f113b6b30bbe8e61a6d564c97e6d8e405a829bcebc0 not found: ID does not exist" containerID="977e6288a8f0b81c6ef15f113b6b30bbe8e61a6d564c97e6d8e405a829bcebc0" Oct 30 13:00:06.662106 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:06.661903 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"977e6288a8f0b81c6ef15f113b6b30bbe8e61a6d564c97e6d8e405a829bcebc0"} err="failed to get container status \"977e6288a8f0b81c6ef15f113b6b30bbe8e61a6d564c97e6d8e405a829bcebc0\": rpc error: code = NotFound desc = could not find container \"977e6288a8f0b81c6ef15f113b6b30bbe8e61a6d564c97e6d8e405a829bcebc0\": container with ID starting with 977e6288a8f0b81c6ef15f113b6b30bbe8e61a6d564c97e6d8e405a829bcebc0 not found: ID does not exist" Oct 30 13:00:06.707491 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:06.707456 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-b6wzm"] Oct 30 13:00:06.719307 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:06.719264 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-b6wzm"] Oct 30 13:00:08.360306 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:00:08.360278 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28420b22-5e86-41ae-b267-bf5688321d66" path="/var/lib/kubelet/pods/28420b22-5e86-41ae-b267-bf5688321d66/volumes" Oct 30 13:00:29.703201 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:00:29.703160 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:00:59.699538 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:00:59.699498 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:01:29.700405 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:01:29.700308 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:01:30.886122 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:30.886089 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr"] Oct 30 13:01:30.886486 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:30.886448 4503 cpu_manager.go:414] "containerMap: removing container" podUID="28420b22-5e86-41ae-b267-bf5688321d66" containerName="manager" Oct 30 13:01:30.886486 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:30.886459 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28420b22-5e86-41ae-b267-bf5688321d66" containerName="manager" Oct 30 13:01:30.886559 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:30.886522 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="28420b22-5e86-41ae-b267-bf5688321d66" containerName="manager" Oct 30 13:01:30.889533 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:30.889514 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:30.907722 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:30.907694 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 13:01:30.917044 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:30.917017 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7g8ld\"" Oct 30 13:01:30.917044 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:30.917038 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 13:01:30.917223 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:30.917049 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 13:01:30.923130 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:30.923107 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr"] Oct 30 13:01:30.949200 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:30.949170 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76757b33-71a1-4186-8ab0-b24bfb08d1c8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr\" (UID: \"76757b33-71a1-4186-8ab0-b24bfb08d1c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:30.949342 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:30.949207 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76757b33-71a1-4186-8ab0-b24bfb08d1c8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr\" (UID: \"76757b33-71a1-4186-8ab0-b24bfb08d1c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:30.949342 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:30.949229 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76757b33-71a1-4186-8ab0-b24bfb08d1c8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr\" (UID: \"76757b33-71a1-4186-8ab0-b24bfb08d1c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:30.949342 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:30.949308 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76757b33-71a1-4186-8ab0-b24bfb08d1c8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr\" (UID: \"76757b33-71a1-4186-8ab0-b24bfb08d1c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:30.949479 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:30.949456 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76757b33-71a1-4186-8ab0-b24bfb08d1c8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr\" (UID: \"76757b33-71a1-4186-8ab0-b24bfb08d1c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:30.949521 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:30.949494 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27r92\" (UniqueName: \"kubernetes.io/projected/76757b33-71a1-4186-8ab0-b24bfb08d1c8-kube-api-access-27r92\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr\" (UID: \"76757b33-71a1-4186-8ab0-b24bfb08d1c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:31.050283 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:31.050251 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27r92\" (UniqueName: \"kubernetes.io/projected/76757b33-71a1-4186-8ab0-b24bfb08d1c8-kube-api-access-27r92\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr\" (UID: \"76757b33-71a1-4186-8ab0-b24bfb08d1c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:31.050462 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:31.050302 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76757b33-71a1-4186-8ab0-b24bfb08d1c8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr\" (UID: \"76757b33-71a1-4186-8ab0-b24bfb08d1c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:31.050462 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:31.050326 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76757b33-71a1-4186-8ab0-b24bfb08d1c8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr\" (UID: \"76757b33-71a1-4186-8ab0-b24bfb08d1c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:31.050462 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:31.050352 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76757b33-71a1-4186-8ab0-b24bfb08d1c8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr\" (UID: \"76757b33-71a1-4186-8ab0-b24bfb08d1c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:31.050462 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:31.050386 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76757b33-71a1-4186-8ab0-b24bfb08d1c8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr\" (UID: \"76757b33-71a1-4186-8ab0-b24bfb08d1c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:31.050462 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:31.050449 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76757b33-71a1-4186-8ab0-b24bfb08d1c8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr\" (UID: \"76757b33-71a1-4186-8ab0-b24bfb08d1c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:31.050768 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:31.050745 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76757b33-71a1-4186-8ab0-b24bfb08d1c8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr\" (UID: \"76757b33-71a1-4186-8ab0-b24bfb08d1c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:31.050847 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:31.050747 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76757b33-71a1-4186-8ab0-b24bfb08d1c8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr\" (UID: \"76757b33-71a1-4186-8ab0-b24bfb08d1c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:31.050847 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:31.050796 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76757b33-71a1-4186-8ab0-b24bfb08d1c8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr\" (UID: \"76757b33-71a1-4186-8ab0-b24bfb08d1c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:31.052887 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:31.052793 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76757b33-71a1-4186-8ab0-b24bfb08d1c8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr\" (UID: \"76757b33-71a1-4186-8ab0-b24bfb08d1c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:31.053058 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:31.052955 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76757b33-71a1-4186-8ab0-b24bfb08d1c8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr\" (UID: \"76757b33-71a1-4186-8ab0-b24bfb08d1c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:31.074423 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:31.074399 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27r92\" (UniqueName: \"kubernetes.io/projected/76757b33-71a1-4186-8ab0-b24bfb08d1c8-kube-api-access-27r92\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr\" (UID: \"76757b33-71a1-4186-8ab0-b24bfb08d1c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:31.202332 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:31.202297 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:31.511750 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:31.511719 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr"] Oct 30 13:01:31.512608 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:01:31.512586 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76757b33_71a1_4186_8ab0_b24bfb08d1c8.slice/crio-f238387efb60ce314cb88b026737bef7afc0eeff9462a404fce9d372214359d4 WatchSource:0}: Error finding container f238387efb60ce314cb88b026737bef7afc0eeff9462a404fce9d372214359d4: Status 404 returned error can't find the container with id f238387efb60ce314cb88b026737bef7afc0eeff9462a404fce9d372214359d4 Oct 30 13:01:32.057215 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:32.057176 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" event={"ID":"76757b33-71a1-4186-8ab0-b24bfb08d1c8","Type":"ContainerStarted","Data":"f238387efb60ce314cb88b026737bef7afc0eeff9462a404fce9d372214359d4"} Oct 30 13:01:33.063648 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.063551 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" event={"ID":"76757b33-71a1-4186-8ab0-b24bfb08d1c8","Type":"ContainerStarted","Data":"6c2ea455686d53843bb4ee5704f5bf7efdaaa8e7e5dab8ac4b8cccd3cdf88056"} Oct 30 13:01:33.119136 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.119085 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" podStartSLOduration=1.889567075 podStartE2EDuration="3.119071647s" totalImagesPullingTime="1.229504572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:33.117221305 +0000 UTC m=+586.940449582" watchObservedRunningTime="2026-10-30 13:01:33.119071647 +0000 UTC m=+586.942299924" Oct 30 13:01:33.203065 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.203034 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:33.391806 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.391734 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk"] Oct 30 13:01:33.401590 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.401563 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:33.418441 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.418404 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk"] Oct 30 13:01:33.427697 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.427674 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 13:01:33.472652 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.472610 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd23c98d-5896-493c-a5f9-b02cd14c3869-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk\" (UID: \"fd23c98d-5896-493c-a5f9-b02cd14c3869\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:33.472847 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.472779 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd23c98d-5896-493c-a5f9-b02cd14c3869-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk\" (UID: \"fd23c98d-5896-493c-a5f9-b02cd14c3869\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:33.472847 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.472831 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd23c98d-5896-493c-a5f9-b02cd14c3869-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk\" (UID: \"fd23c98d-5896-493c-a5f9-b02cd14c3869\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:33.472970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.472906 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd23c98d-5896-493c-a5f9-b02cd14c3869-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk\" (UID: \"fd23c98d-5896-493c-a5f9-b02cd14c3869\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:33.472970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.472961 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd23c98d-5896-493c-a5f9-b02cd14c3869-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk\" (UID: \"fd23c98d-5896-493c-a5f9-b02cd14c3869\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:33.473056 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.472989 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgsqn\" (UniqueName: \"kubernetes.io/projected/fd23c98d-5896-493c-a5f9-b02cd14c3869-kube-api-access-mgsqn\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk\" (UID: \"fd23c98d-5896-493c-a5f9-b02cd14c3869\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:33.573736 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.573693 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd23c98d-5896-493c-a5f9-b02cd14c3869-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk\" (UID: \"fd23c98d-5896-493c-a5f9-b02cd14c3869\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:33.573925 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.573747 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd23c98d-5896-493c-a5f9-b02cd14c3869-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk\" (UID: \"fd23c98d-5896-493c-a5f9-b02cd14c3869\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:33.573925 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.573779 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd23c98d-5896-493c-a5f9-b02cd14c3869-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk\" (UID: \"fd23c98d-5896-493c-a5f9-b02cd14c3869\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:33.573925 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.573797 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgsqn\" (UniqueName: \"kubernetes.io/projected/fd23c98d-5896-493c-a5f9-b02cd14c3869-kube-api-access-mgsqn\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk\" (UID: \"fd23c98d-5896-493c-a5f9-b02cd14c3869\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:33.573925 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.573833 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd23c98d-5896-493c-a5f9-b02cd14c3869-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk\" (UID: \"fd23c98d-5896-493c-a5f9-b02cd14c3869\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:33.573925 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.573911 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd23c98d-5896-493c-a5f9-b02cd14c3869-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk\" (UID: \"fd23c98d-5896-493c-a5f9-b02cd14c3869\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:33.574393 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.574367 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd23c98d-5896-493c-a5f9-b02cd14c3869-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk\" (UID: \"fd23c98d-5896-493c-a5f9-b02cd14c3869\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:33.574518 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.574500 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd23c98d-5896-493c-a5f9-b02cd14c3869-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk\" (UID: \"fd23c98d-5896-493c-a5f9-b02cd14c3869\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:33.574768 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.574672 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd23c98d-5896-493c-a5f9-b02cd14c3869-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk\" (UID: \"fd23c98d-5896-493c-a5f9-b02cd14c3869\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:33.576387 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.576367 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd23c98d-5896-493c-a5f9-b02cd14c3869-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk\" (UID: \"fd23c98d-5896-493c-a5f9-b02cd14c3869\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:33.576558 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.576540 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd23c98d-5896-493c-a5f9-b02cd14c3869-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk\" (UID: \"fd23c98d-5896-493c-a5f9-b02cd14c3869\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:33.617740 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.617708 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgsqn\" (UniqueName: \"kubernetes.io/projected/fd23c98d-5896-493c-a5f9-b02cd14c3869-kube-api-access-mgsqn\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk\" (UID: \"fd23c98d-5896-493c-a5f9-b02cd14c3869\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:33.714649 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:33.714610 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:34.021799 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:34.021766 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk"] Oct 30 13:01:34.022436 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:01:34.022397 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd23c98d_5896_493c_a5f9_b02cd14c3869.slice/crio-cc0a7fdbd209a9af50d1ea4cbe2266edf59992adbcbcdcd978f9e904ffeafff8 WatchSource:0}: Error finding container cc0a7fdbd209a9af50d1ea4cbe2266edf59992adbcbcdcd978f9e904ffeafff8: Status 404 returned error can't find the container with id cc0a7fdbd209a9af50d1ea4cbe2266edf59992adbcbcdcd978f9e904ffeafff8 Oct 30 13:01:34.069817 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:34.069781 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" event={"ID":"fd23c98d-5896-493c-a5f9-b02cd14c3869","Type":"ContainerStarted","Data":"cc0a7fdbd209a9af50d1ea4cbe2266edf59992adbcbcdcd978f9e904ffeafff8"} Oct 30 13:01:35.076613 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:35.076573 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" event={"ID":"fd23c98d-5896-493c-a5f9-b02cd14c3869","Type":"ContainerStarted","Data":"a435d4c7d7372d13b4f33fbcd5470232b6bc0f9be849e61feabc76fd26696af3"} Oct 30 13:01:35.125152 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:35.125102 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" podStartSLOduration=1.93382966 podStartE2EDuration="2.125084412s" totalImagesPullingTime="191.254752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:35.124768043 +0000 UTC m=+588.947996313" watchObservedRunningTime="2026-10-30 13:01:35.125084412 +0000 UTC m=+588.948312670" Oct 30 13:01:35.715499 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:35.715464 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:37.023293 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.023254 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf"] Oct 30 13:01:37.027408 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.027389 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.048255 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.048225 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf"] Oct 30 13:01:37.058218 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.058192 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 13:01:37.107989 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.107959 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9e69714-f702-4f5b-9fdf-a1333076beb4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf\" (UID: \"a9e69714-f702-4f5b-9fdf-a1333076beb4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.108162 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.108021 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9e69714-f702-4f5b-9fdf-a1333076beb4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf\" (UID: \"a9e69714-f702-4f5b-9fdf-a1333076beb4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.108162 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.108076 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9e69714-f702-4f5b-9fdf-a1333076beb4-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf\" (UID: \"a9e69714-f702-4f5b-9fdf-a1333076beb4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.108251 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.108178 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rdrn\" (UniqueName: \"kubernetes.io/projected/a9e69714-f702-4f5b-9fdf-a1333076beb4-kube-api-access-5rdrn\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf\" (UID: \"a9e69714-f702-4f5b-9fdf-a1333076beb4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.108251 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.108212 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9e69714-f702-4f5b-9fdf-a1333076beb4-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf\" (UID: \"a9e69714-f702-4f5b-9fdf-a1333076beb4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.108251 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.108234 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9e69714-f702-4f5b-9fdf-a1333076beb4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf\" (UID: \"a9e69714-f702-4f5b-9fdf-a1333076beb4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.209239 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.209199 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9e69714-f702-4f5b-9fdf-a1333076beb4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf\" (UID: \"a9e69714-f702-4f5b-9fdf-a1333076beb4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.209433 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.209255 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9e69714-f702-4f5b-9fdf-a1333076beb4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf\" (UID: \"a9e69714-f702-4f5b-9fdf-a1333076beb4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.209433 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.209300 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9e69714-f702-4f5b-9fdf-a1333076beb4-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf\" (UID: \"a9e69714-f702-4f5b-9fdf-a1333076beb4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.209433 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.209345 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rdrn\" (UniqueName: \"kubernetes.io/projected/a9e69714-f702-4f5b-9fdf-a1333076beb4-kube-api-access-5rdrn\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf\" (UID: \"a9e69714-f702-4f5b-9fdf-a1333076beb4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.209433 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.209379 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9e69714-f702-4f5b-9fdf-a1333076beb4-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf\" (UID: \"a9e69714-f702-4f5b-9fdf-a1333076beb4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.209433 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.209411 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9e69714-f702-4f5b-9fdf-a1333076beb4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf\" (UID: \"a9e69714-f702-4f5b-9fdf-a1333076beb4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.209750 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.209729 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9e69714-f702-4f5b-9fdf-a1333076beb4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf\" (UID: \"a9e69714-f702-4f5b-9fdf-a1333076beb4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.209825 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.209770 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9e69714-f702-4f5b-9fdf-a1333076beb4-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf\" (UID: \"a9e69714-f702-4f5b-9fdf-a1333076beb4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.209825 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.209792 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9e69714-f702-4f5b-9fdf-a1333076beb4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf\" (UID: \"a9e69714-f702-4f5b-9fdf-a1333076beb4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.212077 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.212048 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9e69714-f702-4f5b-9fdf-a1333076beb4-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf\" (UID: \"a9e69714-f702-4f5b-9fdf-a1333076beb4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.212357 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.212340 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9e69714-f702-4f5b-9fdf-a1333076beb4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf\" (UID: \"a9e69714-f702-4f5b-9fdf-a1333076beb4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.235400 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.235373 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rdrn\" (UniqueName: \"kubernetes.io/projected/a9e69714-f702-4f5b-9fdf-a1333076beb4-kube-api-access-5rdrn\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf\" (UID: \"a9e69714-f702-4f5b-9fdf-a1333076beb4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.341671 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.341592 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:37.653194 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:37.653163 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf"] Oct 30 13:01:37.653312 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:01:37.653265 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e69714_f702_4f5b_9fdf_a1333076beb4.slice/crio-c32005e06118b277c4a9c8b0d8dde86c96758c9cc0a97d9f0b5aabf855d8de9c WatchSource:0}: Error finding container c32005e06118b277c4a9c8b0d8dde86c96758c9cc0a97d9f0b5aabf855d8de9c: Status 404 returned error can't find the container with id c32005e06118b277c4a9c8b0d8dde86c96758c9cc0a97d9f0b5aabf855d8de9c Oct 30 13:01:38.098768 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:38.098732 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" event={"ID":"a9e69714-f702-4f5b-9fdf-a1333076beb4","Type":"ContainerStarted","Data":"4783db6c0be0df4f9ee51cd07a711f636a037b3b0951e762b85cd32138a74aae"} Oct 30 13:01:38.099165 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:38.098810 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" event={"ID":"a9e69714-f702-4f5b-9fdf-a1333076beb4","Type":"ContainerStarted","Data":"c32005e06118b277c4a9c8b0d8dde86c96758c9cc0a97d9f0b5aabf855d8de9c"} Oct 30 13:01:38.143802 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:38.143744 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" podStartSLOduration=1.95805582 podStartE2EDuration="2.143729535s" totalImagesPullingTime="185.673715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:38.142339721 +0000 UTC m=+591.965568001" watchObservedRunningTime="2026-10-30 13:01:38.143729535 +0000 UTC m=+591.966957811" Oct 30 13:01:38.342061 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:38.341972 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:39.072405 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.072375 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx"] Oct 30 13:01:39.076540 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.076515 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.106015 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.105979 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 13:01:39.108283 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.108258 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx"] Oct 30 13:01:39.127104 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.127069 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk5kq\" (UniqueName: \"kubernetes.io/projected/5522db7b-cfb1-4355-bbe0-b5022eac5e82-kube-api-access-kk5kq\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7tjtx\" (UID: \"5522db7b-cfb1-4355-bbe0-b5022eac5e82\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.127289 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.127134 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5522db7b-cfb1-4355-bbe0-b5022eac5e82-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7tjtx\" (UID: \"5522db7b-cfb1-4355-bbe0-b5022eac5e82\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.127289 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.127242 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5522db7b-cfb1-4355-bbe0-b5022eac5e82-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7tjtx\" (UID: \"5522db7b-cfb1-4355-bbe0-b5022eac5e82\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.127289 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.127276 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5522db7b-cfb1-4355-bbe0-b5022eac5e82-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7tjtx\" (UID: \"5522db7b-cfb1-4355-bbe0-b5022eac5e82\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.127468 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.127336 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5522db7b-cfb1-4355-bbe0-b5022eac5e82-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7tjtx\" (UID: \"5522db7b-cfb1-4355-bbe0-b5022eac5e82\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.127468 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.127413 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5522db7b-cfb1-4355-bbe0-b5022eac5e82-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7tjtx\" (UID: \"5522db7b-cfb1-4355-bbe0-b5022eac5e82\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.228394 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.228353 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5522db7b-cfb1-4355-bbe0-b5022eac5e82-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7tjtx\" (UID: \"5522db7b-cfb1-4355-bbe0-b5022eac5e82\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.228579 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.228548 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5522db7b-cfb1-4355-bbe0-b5022eac5e82-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7tjtx\" (UID: \"5522db7b-cfb1-4355-bbe0-b5022eac5e82\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.228645 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.228611 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk5kq\" (UniqueName: \"kubernetes.io/projected/5522db7b-cfb1-4355-bbe0-b5022eac5e82-kube-api-access-kk5kq\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7tjtx\" (UID: \"5522db7b-cfb1-4355-bbe0-b5022eac5e82\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.228711 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.228661 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5522db7b-cfb1-4355-bbe0-b5022eac5e82-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7tjtx\" (UID: \"5522db7b-cfb1-4355-bbe0-b5022eac5e82\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.228711 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.228705 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5522db7b-cfb1-4355-bbe0-b5022eac5e82-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7tjtx\" (UID: \"5522db7b-cfb1-4355-bbe0-b5022eac5e82\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.228816 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.228735 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5522db7b-cfb1-4355-bbe0-b5022eac5e82-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7tjtx\" (UID: \"5522db7b-cfb1-4355-bbe0-b5022eac5e82\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.228816 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.228742 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5522db7b-cfb1-4355-bbe0-b5022eac5e82-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7tjtx\" (UID: \"5522db7b-cfb1-4355-bbe0-b5022eac5e82\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.229024 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.229001 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5522db7b-cfb1-4355-bbe0-b5022eac5e82-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7tjtx\" (UID: \"5522db7b-cfb1-4355-bbe0-b5022eac5e82\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.229107 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.229041 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5522db7b-cfb1-4355-bbe0-b5022eac5e82-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7tjtx\" (UID: \"5522db7b-cfb1-4355-bbe0-b5022eac5e82\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.230932 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.230908 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5522db7b-cfb1-4355-bbe0-b5022eac5e82-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7tjtx\" (UID: \"5522db7b-cfb1-4355-bbe0-b5022eac5e82\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.231231 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.231214 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5522db7b-cfb1-4355-bbe0-b5022eac5e82-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7tjtx\" (UID: \"5522db7b-cfb1-4355-bbe0-b5022eac5e82\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.253967 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.253912 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk5kq\" (UniqueName: \"kubernetes.io/projected/5522db7b-cfb1-4355-bbe0-b5022eac5e82-kube-api-access-kk5kq\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7tjtx\" (UID: \"5522db7b-cfb1-4355-bbe0-b5022eac5e82\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.391118 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.391038 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:39.713532 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:39.713500 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx"] Oct 30 13:01:39.713975 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:01:39.713924 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5522db7b_cfb1_4355_bbe0_b5022eac5e82.slice/crio-938e86dd6e454c8b39156e6fc56fc1962397de6494d3277b60e3adedf3ed9c1c WatchSource:0}: Error finding container 938e86dd6e454c8b39156e6fc56fc1962397de6494d3277b60e3adedf3ed9c1c: Status 404 returned error can't find the container with id 938e86dd6e454c8b39156e6fc56fc1962397de6494d3277b60e3adedf3ed9c1c Oct 30 13:01:40.112391 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:40.112341 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" event={"ID":"5522db7b-cfb1-4355-bbe0-b5022eac5e82","Type":"ContainerStarted","Data":"bbc4ebfd8d3431087646d7b3e7f3afebf430b3454f2b2e545741c60a16ce8965"} Oct 30 13:01:40.112874 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:40.112435 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" event={"ID":"5522db7b-cfb1-4355-bbe0-b5022eac5e82","Type":"ContainerStarted","Data":"938e86dd6e454c8b39156e6fc56fc1962397de6494d3277b60e3adedf3ed9c1c"} Oct 30 13:01:40.164128 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:40.159659 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" podStartSLOduration=0.938509355 podStartE2EDuration="1.15964109s" totalImagesPullingTime="221.131735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:40.158544065 +0000 UTC m=+593.981772342" watchObservedRunningTime="2026-10-30 13:01:40.15964109 +0000 UTC m=+593.982869368" Oct 30 13:01:40.391631 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:40.391539 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:41.202981 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:41.202918 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:41.215890 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:41.215859 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:42.134816 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.134785 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" Oct 30 13:01:42.589683 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.589648 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p"] Oct 30 13:01:42.592801 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.592778 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:42.609265 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.609229 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 13:01:42.623072 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.623044 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p"] Oct 30 13:01:42.660022 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.659919 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g9g7p\" (UID: \"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:42.660168 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.660039 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g9g7p\" (UID: \"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:42.660168 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.660058 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwctl\" (UniqueName: \"kubernetes.io/projected/0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf-kube-api-access-gwctl\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g9g7p\" (UID: \"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:42.660168 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.660087 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g9g7p\" (UID: \"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:42.660168 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.660117 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g9g7p\" (UID: \"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:42.660312 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.660227 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g9g7p\" (UID: \"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:42.761014 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.760974 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g9g7p\" (UID: \"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:42.761201 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.761032 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g9g7p\" (UID: \"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:42.761201 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.761072 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwctl\" (UniqueName: \"kubernetes.io/projected/0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf-kube-api-access-gwctl\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g9g7p\" (UID: \"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:42.761201 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.761122 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g9g7p\" (UID: \"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:42.761201 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.761175 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g9g7p\" (UID: \"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:42.761435 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.761212 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g9g7p\" (UID: \"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:42.761490 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.761433 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g9g7p\" (UID: \"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:42.761798 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.761767 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g9g7p\" (UID: \"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:42.761798 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.761786 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g9g7p\" (UID: \"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:42.763957 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.763918 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g9g7p\" (UID: \"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:42.764214 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.764195 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g9g7p\" (UID: \"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:42.785548 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.785524 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwctl\" (UniqueName: \"kubernetes.io/projected/0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf-kube-api-access-gwctl\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g9g7p\" (UID: \"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:42.907163 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:42.907124 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:43.207804 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:43.207770 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p"] Oct 30 13:01:43.208510 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:01:43.208482 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf5cc67_e95c_4cf2_95dc_4ea9697fb4bf.slice/crio-60ca8486c4ffe0397bc3a1454cab5cbab935be4d07f7b08f869349a8cebdead0 WatchSource:0}: Error finding container 60ca8486c4ffe0397bc3a1454cab5cbab935be4d07f7b08f869349a8cebdead0: Status 404 returned error can't find the container with id 60ca8486c4ffe0397bc3a1454cab5cbab935be4d07f7b08f869349a8cebdead0 Oct 30 13:01:43.715638 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:43.715559 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:43.728388 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:43.728363 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:44.134268 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:44.134170 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" event={"ID":"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf","Type":"ContainerStarted","Data":"9d6c218027d427b7b2b0c5d30c459efbadd5876e895d3c6eb3b71d0c2ada0ca7"} Oct 30 13:01:44.134268 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:44.134238 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" event={"ID":"0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf","Type":"ContainerStarted","Data":"60ca8486c4ffe0397bc3a1454cab5cbab935be4d07f7b08f869349a8cebdead0"} Oct 30 13:01:44.145429 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:44.145400 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk" Oct 30 13:01:44.178950 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:44.178876 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" podStartSLOduration=1.980480151 podStartE2EDuration="2.178856222s" totalImagesPullingTime="198.376071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:44.176715112 +0000 UTC m=+597.999943411" watchObservedRunningTime="2026-10-30 13:01:44.178856222 +0000 UTC m=+598.002084503" Oct 30 13:01:44.908044 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:44.908007 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:47.342336 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:47.342295 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:47.355167 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:47.355139 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:48.171327 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:48.171296 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf" Oct 30 13:01:49.391429 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:49.391385 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:49.404244 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:49.404213 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:50.176470 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:50.176437 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" Oct 30 13:01:51.555404 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:51.555364 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-p2bnv"] Oct 30 13:01:51.560161 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:51.560139 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-p2bnv" Oct 30 13:01:51.585861 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:51.585828 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-p2bnv"] Oct 30 13:01:51.594445 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:51.594408 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 13:01:51.594445 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:51.594411 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 13:01:51.594682 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:51.594569 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-m7t6c\"" Oct 30 13:01:51.662101 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:51.661664 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b28w9\" (UniqueName: \"kubernetes.io/projected/0c73a484-f195-46ce-9856-b754992272db-kube-api-access-b28w9\") pod \"ai-gateway-controller-697b76b867-p2bnv\" (UID: \"0c73a484-f195-46ce-9856-b754992272db\") " pod="opendatahub/ai-gateway-controller-697b76b867-p2bnv" Oct 30 13:01:51.662101 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:51.661720 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c73a484-f195-46ce-9856-b754992272db-metrics-certs\") pod \"ai-gateway-controller-697b76b867-p2bnv\" (UID: \"0c73a484-f195-46ce-9856-b754992272db\") " pod="opendatahub/ai-gateway-controller-697b76b867-p2bnv" Oct 30 13:01:51.763164 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:51.763120 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b28w9\" (UniqueName: \"kubernetes.io/projected/0c73a484-f195-46ce-9856-b754992272db-kube-api-access-b28w9\") pod \"ai-gateway-controller-697b76b867-p2bnv\" (UID: \"0c73a484-f195-46ce-9856-b754992272db\") " pod="opendatahub/ai-gateway-controller-697b76b867-p2bnv" Oct 30 13:01:51.763343 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:51.763172 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c73a484-f195-46ce-9856-b754992272db-metrics-certs\") pod \"ai-gateway-controller-697b76b867-p2bnv\" (UID: \"0c73a484-f195-46ce-9856-b754992272db\") " pod="opendatahub/ai-gateway-controller-697b76b867-p2bnv" Oct 30 13:01:51.765625 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:51.765604 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c73a484-f195-46ce-9856-b754992272db-metrics-certs\") pod \"ai-gateway-controller-697b76b867-p2bnv\" (UID: \"0c73a484-f195-46ce-9856-b754992272db\") " pod="opendatahub/ai-gateway-controller-697b76b867-p2bnv" Oct 30 13:01:51.791015 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:51.790976 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b28w9\" (UniqueName: \"kubernetes.io/projected/0c73a484-f195-46ce-9856-b754992272db-kube-api-access-b28w9\") pod \"ai-gateway-controller-697b76b867-p2bnv\" (UID: \"0c73a484-f195-46ce-9856-b754992272db\") " pod="opendatahub/ai-gateway-controller-697b76b867-p2bnv" Oct 30 13:01:51.874526 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:51.874431 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-p2bnv" Oct 30 13:01:52.197867 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:52.197833 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-p2bnv"] Oct 30 13:01:52.198844 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:01:52.198815 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c73a484_f195_46ce_9856_b754992272db.slice/crio-9810da97fbe06b5834abeaa8d00c9751032f7845da9d55e8b28bf0280d3b9116 WatchSource:0}: Error finding container 9810da97fbe06b5834abeaa8d00c9751032f7845da9d55e8b28bf0280d3b9116: Status 404 returned error can't find the container with id 9810da97fbe06b5834abeaa8d00c9751032f7845da9d55e8b28bf0280d3b9116 Oct 30 13:01:52.908192 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:52.907536 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:52.924675 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:52.924416 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:53.185487 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:53.185405 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-p2bnv" event={"ID":"0c73a484-f195-46ce-9856-b754992272db","Type":"ContainerStarted","Data":"9810da97fbe06b5834abeaa8d00c9751032f7845da9d55e8b28bf0280d3b9116"} Oct 30 13:01:53.202933 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:53.202900 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g9g7p" Oct 30 13:01:55.198010 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:55.197965 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-p2bnv" event={"ID":"0c73a484-f195-46ce-9856-b754992272db","Type":"ContainerStarted","Data":"7f96b58e4744560a65377777ae4860cd186a001e89bba06effc2f833859af75e"} Oct 30 13:01:55.198430 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:55.198041 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-697b76b867-p2bnv" Oct 30 13:01:55.240481 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:01:55.240421 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-697b76b867-p2bnv" podStartSLOduration=1.816761628 podStartE2EDuration="4.240407111s" totalImagesPullingTime="2.423645483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:55.238773773 +0000 UTC m=+609.062002063" watchObservedRunningTime="2026-10-30 13:01:55.240407111 +0000 UTC m=+609.063635385" Oct 30 13:01:59.701176 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:01:59.701138 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:02:06.206005 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:06.205974 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-697b76b867-p2bnv" Oct 30 13:02:08.650658 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.650618 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx"] Oct 30 13:02:08.654511 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.654488 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.664284 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.664260 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4"] Oct 30 13:02:08.667564 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.667545 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.681703 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.681631 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx"] Oct 30 13:02:08.686431 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.686397 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4"] Oct 30 13:02:08.708807 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.708766 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec8b24c-a530-4514-9f0b-5a11af7776ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.708985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.708818 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8daeac71-175e-4684-84df-b27b0d52e733-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.708985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.708931 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.708985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.708968 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.709165 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.708987 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8daeac71-175e-4684-84df-b27b0d52e733-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.709165 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.709037 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5mfw\" (UniqueName: \"kubernetes.io/projected/bec8b24c-a530-4514-9f0b-5a11af7776ce-kube-api-access-v5mfw\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.709165 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.709077 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.709165 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.709108 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.709165 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.709135 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.709165 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.709160 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.709439 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.709228 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.709439 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.709266 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpx2g\" (UniqueName: \"kubernetes.io/projected/8daeac71-175e-4684-84df-b27b0d52e733-kube-api-access-tpx2g\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.709439 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.709295 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.709439 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.709311 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.709439 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.709328 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.709439 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.709391 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.709439 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.709432 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.709689 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.709468 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8daeac71-175e-4684-84df-b27b0d52e733-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.758839 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.758803 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-g8svl\"" Oct 30 13:02:08.810446 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.810405 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.810627 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.810451 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpx2g\" (UniqueName: \"kubernetes.io/projected/8daeac71-175e-4684-84df-b27b0d52e733-kube-api-access-tpx2g\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.810627 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.810502 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.810627 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.810527 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.810627 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.810556 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.810627 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.810587 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.810627 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.810615 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.811009 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.810647 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8daeac71-175e-4684-84df-b27b0d52e733-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.811009 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.810698 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec8b24c-a530-4514-9f0b-5a11af7776ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.811009 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.810727 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8daeac71-175e-4684-84df-b27b0d52e733-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.811009 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.810793 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.811009 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.810833 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.811009 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.810859 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8daeac71-175e-4684-84df-b27b0d52e733-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.811009 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.810901 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.811400 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.811009 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.811400 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.811048 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.811400 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.811067 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.811400 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.811149 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5mfw\" (UniqueName: \"kubernetes.io/projected/bec8b24c-a530-4514-9f0b-5a11af7776ce-kube-api-access-v5mfw\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.811400 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.811185 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.811400 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.811215 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.811400 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.811244 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.811400 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.811272 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.811400 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.811379 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.811915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.811601 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8daeac71-175e-4684-84df-b27b0d52e733-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.811915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.811603 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.811915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.811658 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.811915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.811665 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec8b24c-a530-4514-9f0b-5a11af7776ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.811915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.811829 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.813310 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.813286 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.813528 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.813511 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.813528 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.813521 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8daeac71-175e-4684-84df-b27b0d52e733-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.813719 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.813703 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.836336 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.836304 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8daeac71-175e-4684-84df-b27b0d52e733-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.836466 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.836424 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.836674 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.836645 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpx2g\" (UniqueName: \"kubernetes.io/projected/8daeac71-175e-4684-84df-b27b0d52e733-kube-api-access-tpx2g\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zxmjx\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.837293 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.837272 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5mfw\" (UniqueName: \"kubernetes.io/projected/bec8b24c-a530-4514-9f0b-5a11af7776ce-kube-api-access-v5mfw\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nm7s4\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:08.969151 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.969062 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:08.980485 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:08.980465 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:09.300672 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:09.300638 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx"] Oct 30 13:02:09.301231 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:02:09.301202 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8daeac71_175e_4684_84df_b27b0d52e733.slice/crio-cd2c183b0f6e0a5038eadaac2be3026ecd4c85b5a83d4bc79ed153d9fbab1e6e WatchSource:0}: Error finding container cd2c183b0f6e0a5038eadaac2be3026ecd4c85b5a83d4bc79ed153d9fbab1e6e: Status 404 returned error can't find the container with id cd2c183b0f6e0a5038eadaac2be3026ecd4c85b5a83d4bc79ed153d9fbab1e6e Oct 30 13:02:09.302911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:09.302862 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4"] Oct 30 13:02:09.311118 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:02:09.311093 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec8b24c_a530_4514_9f0b_5a11af7776ce.slice/crio-1a7ea3d3562fa051f8db10a54c1161534bcab611530fed97d06af69bc9005998 WatchSource:0}: Error finding container 1a7ea3d3562fa051f8db10a54c1161534bcab611530fed97d06af69bc9005998: Status 404 returned error can't find the container with id 1a7ea3d3562fa051f8db10a54c1161534bcab611530fed97d06af69bc9005998 Oct 30 13:02:10.280378 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:10.280338 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" event={"ID":"bec8b24c-a530-4514-9f0b-5a11af7776ce","Type":"ContainerStarted","Data":"1a7ea3d3562fa051f8db10a54c1161534bcab611530fed97d06af69bc9005998"} Oct 30 13:02:10.281849 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:10.281819 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" event={"ID":"8daeac71-175e-4684-84df-b27b0d52e733","Type":"ContainerStarted","Data":"cd2c183b0f6e0a5038eadaac2be3026ecd4c85b5a83d4bc79ed153d9fbab1e6e"} Oct 30 13:02:12.317782 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:12.317748 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:12.318184 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:12.317833 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:12.318184 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:12.317883 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:12.326573 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:12.326521 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:12.326661 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:12.326602 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:12.326661 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:12.326639 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:13.312063 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:13.312020 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" event={"ID":"bec8b24c-a530-4514-9f0b-5a11af7776ce","Type":"ContainerStarted","Data":"7398c73c609808de1e473d0405e9e5c221e59ad2e2c2cd09544fd166387362e3"} Oct 30 13:02:13.314078 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:13.314050 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" event={"ID":"8daeac71-175e-4684-84df-b27b0d52e733","Type":"ContainerStarted","Data":"2a3dbc6b593fa6b5f6af8b9510907704a87be349f9a8c5fd2219811b1ab2faf0"} Oct 30 13:02:13.388964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:13.373791 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" podStartSLOduration=2.36106931 podStartE2EDuration="5.373771542s" totalImagesPullingTime="3.012702232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:13.370569021 +0000 UTC m=+627.193797321" watchObservedRunningTime="2026-10-30 13:02:13.373771542 +0000 UTC m=+627.196999820" Oct 30 13:02:13.423990 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:13.423921 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" podStartSLOduration=2.4101949830000002 podStartE2EDuration="5.423907737s" totalImagesPullingTime="3.013712754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:13.421464529 +0000 UTC m=+627.244692817" watchObservedRunningTime="2026-10-30 13:02:13.423907737 +0000 UTC m=+627.247136076" Oct 30 13:02:13.973965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:13.973788 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:13.974555 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:13.973921 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zxmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 13:02:13.974555 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:13.974236 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" podUID="8daeac71-175e-4684-84df-b27b0d52e733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 13:02:13.984530 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:13.984290 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:13.984530 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:13.984397 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-nm7s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 13:02:13.984530 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:13.984473 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" podUID="bec8b24c-a530-4514-9f0b-5a11af7776ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 13:02:14.972965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:14.970373 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zxmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 13:02:14.972965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:14.970436 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" podUID="8daeac71-175e-4684-84df-b27b0d52e733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 13:02:14.984964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:14.983319 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-nm7s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 13:02:14.984964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:14.983380 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" podUID="bec8b24c-a530-4514-9f0b-5a11af7776ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 13:02:15.976975 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:15.976398 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zxmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 13:02:15.976975 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:15.976482 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" podUID="8daeac71-175e-4684-84df-b27b0d52e733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 13:02:15.984962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:15.983311 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-nm7s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 13:02:15.984962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:15.983383 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" podUID="bec8b24c-a530-4514-9f0b-5a11af7776ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 13:02:16.972442 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:16.972406 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zxmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 13:02:16.972730 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:16.972702 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" podUID="8daeac71-175e-4684-84df-b27b0d52e733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 13:02:16.981280 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:16.981254 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-nm7s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 13:02:16.981768 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:16.981744 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" podUID="bec8b24c-a530-4514-9f0b-5a11af7776ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 13:02:17.975964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:17.975400 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zxmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 13:02:17.975964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:17.975471 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" podUID="8daeac71-175e-4684-84df-b27b0d52e733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 13:02:17.983964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:17.982314 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-nm7s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 13:02:17.983964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:17.982373 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" podUID="bec8b24c-a530-4514-9f0b-5a11af7776ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 13:02:18.970257 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:18.970217 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zxmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 13:02:18.970463 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:18.970293 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" podUID="8daeac71-175e-4684-84df-b27b0d52e733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 13:02:18.981514 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:18.981482 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-nm7s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 13:02:18.981633 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:18.981548 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" podUID="bec8b24c-a530-4514-9f0b-5a11af7776ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 13:02:20.709321 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:20.709287 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:20.710435 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:20.710408 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:20.711346 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:20.711325 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:02:20.740520 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:20.740497 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:20.740882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:20.740856 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:20.741668 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:20.741648 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:02:21.090794 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:21.090704 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-p2bnv"] Oct 30 13:02:21.091002 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:21.090972 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-697b76b867-p2bnv" podUID="0c73a484-f195-46ce-9856-b754992272db" containerName="manager" containerID="cri-o://7f96b58e4744560a65377777ae4860cd186a001e89bba06effc2f833859af75e" gracePeriod=30 Oct 30 13:02:21.346305 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:21.346245 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-p2bnv" Oct 30 13:02:21.366875 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:21.366848 4503 generic.go:356] "Generic (PLEG): container finished" podID="0c73a484-f195-46ce-9856-b754992272db" containerID="7f96b58e4744560a65377777ae4860cd186a001e89bba06effc2f833859af75e" exitCode=0 Oct 30 13:02:21.367034 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:21.366892 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-p2bnv" Oct 30 13:02:21.367034 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:21.366958 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-p2bnv" event={"ID":"0c73a484-f195-46ce-9856-b754992272db","Type":"ContainerDied","Data":"7f96b58e4744560a65377777ae4860cd186a001e89bba06effc2f833859af75e"} Oct 30 13:02:21.367034 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:21.367022 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-p2bnv" event={"ID":"0c73a484-f195-46ce-9856-b754992272db","Type":"ContainerDied","Data":"9810da97fbe06b5834abeaa8d00c9751032f7845da9d55e8b28bf0280d3b9116"} Oct 30 13:02:21.367198 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:21.367056 4503 scope.go:122] "RemoveContainer" containerID="7f96b58e4744560a65377777ae4860cd186a001e89bba06effc2f833859af75e" Oct 30 13:02:21.375848 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:21.375831 4503 scope.go:122] "RemoveContainer" containerID="7f96b58e4744560a65377777ae4860cd186a001e89bba06effc2f833859af75e" Oct 30 13:02:21.376193 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:02:21.376174 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f96b58e4744560a65377777ae4860cd186a001e89bba06effc2f833859af75e\": container with ID starting with 7f96b58e4744560a65377777ae4860cd186a001e89bba06effc2f833859af75e not found: ID does not exist" containerID="7f96b58e4744560a65377777ae4860cd186a001e89bba06effc2f833859af75e" Oct 30 13:02:21.376249 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:21.376203 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f96b58e4744560a65377777ae4860cd186a001e89bba06effc2f833859af75e"} err="failed to get container status \"7f96b58e4744560a65377777ae4860cd186a001e89bba06effc2f833859af75e\": rpc error: code = NotFound desc = could not find container \"7f96b58e4744560a65377777ae4860cd186a001e89bba06effc2f833859af75e\": container with ID starting with 7f96b58e4744560a65377777ae4860cd186a001e89bba06effc2f833859af75e not found: ID does not exist" Oct 30 13:02:21.393439 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:21.393417 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c73a484-f195-46ce-9856-b754992272db-kube-api-access-b28w9\" (UniqueName: \"kubernetes.io/projected/0c73a484-f195-46ce-9856-b754992272db-kube-api-access-b28w9\") pod \"0c73a484-f195-46ce-9856-b754992272db\" (UID: \"0c73a484-f195-46ce-9856-b754992272db\") " Oct 30 13:02:21.393535 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:21.393470 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c73a484-f195-46ce-9856-b754992272db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c73a484-f195-46ce-9856-b754992272db-metrics-certs\") pod \"0c73a484-f195-46ce-9856-b754992272db\" (UID: \"0c73a484-f195-46ce-9856-b754992272db\") " Oct 30 13:02:21.395736 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:21.395713 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c73a484-f195-46ce-9856-b754992272db-kube-api-access-b28w9" pod "0c73a484-f195-46ce-9856-b754992272db" (UID: "0c73a484-f195-46ce-9856-b754992272db"). InnerVolumeSpecName "kube-api-access-b28w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:21.396014 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:21.395983 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c73a484-f195-46ce-9856-b754992272db-metrics-certs" pod "0c73a484-f195-46ce-9856-b754992272db" (UID: "0c73a484-f195-46ce-9856-b754992272db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:02:21.495153 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:21.495118 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b28w9\" (UniqueName: \"kubernetes.io/projected/0c73a484-f195-46ce-9856-b754992272db-kube-api-access-b28w9\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:02:21.495153 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:21.495152 4503 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c73a484-f195-46ce-9856-b754992272db-metrics-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:02:21.724503 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:21.724467 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-p2bnv"] Oct 30 13:02:21.741910 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:21.741872 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-p2bnv"] Oct 30 13:02:22.364092 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:22.361879 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c73a484-f195-46ce-9856-b754992272db" path="/var/lib/kubelet/pods/0c73a484-f195-46ce-9856-b754992272db/volumes" Oct 30 13:02:23.711096 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.711063 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd"] Oct 30 13:02:23.712208 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.712180 4503 cpu_manager.go:414] "containerMap: removing container" podUID="0c73a484-f195-46ce-9856-b754992272db" containerName="manager" Oct 30 13:02:23.712208 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.712206 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c73a484-f195-46ce-9856-b754992272db" containerName="manager" Oct 30 13:02:23.712420 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.712311 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c73a484-f195-46ce-9856-b754992272db" containerName="manager" Oct 30 13:02:23.717032 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.717007 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.747607 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.747574 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd"] Oct 30 13:02:23.816790 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.816757 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.816974 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.816806 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.816974 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.816914 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.817059 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.817026 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.817112 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.817056 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.817112 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.817074 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.817217 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.817138 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.817217 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.817195 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p5qb\" (UniqueName: \"kubernetes.io/projected/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-kube-api-access-8p5qb\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.817307 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.817288 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.918515 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.918479 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.918515 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.918515 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.918747 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.918531 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.918747 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.918551 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.918747 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.918613 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p5qb\" (UniqueName: \"kubernetes.io/projected/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-kube-api-access-8p5qb\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.918747 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.918655 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.918747 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.918719 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.919040 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.918753 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.919040 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.918796 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.919040 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.918896 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.919209 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.919097 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.919209 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.919183 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.919280 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.919210 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.919495 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.919474 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.920877 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.920850 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.921021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.921003 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.945800 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.945763 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:23.945972 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:23.945955 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p5qb\" (UniqueName: \"kubernetes.io/projected/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-kube-api-access-8p5qb\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qvxgd\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:24.033461 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:24.033384 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:24.348150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:24.348126 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd"] Oct 30 13:02:24.349378 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:02:24.349349 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a6bc11_e10e_42aa_a182_38c19cbc9dcf.slice/crio-ee595bd35c57acf192d59cf42d4dea0dee8c7b5b799a16e8d05945d6dd993eac WatchSource:0}: Error finding container ee595bd35c57acf192d59cf42d4dea0dee8c7b5b799a16e8d05945d6dd993eac: Status 404 returned error can't find the container with id ee595bd35c57acf192d59cf42d4dea0dee8c7b5b799a16e8d05945d6dd993eac Oct 30 13:02:24.351472 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:24.351442 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:24.351533 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:24.351504 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:24.351567 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:24.351531 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:24.382977 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:24.382930 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" event={"ID":"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf","Type":"ContainerStarted","Data":"ee595bd35c57acf192d59cf42d4dea0dee8c7b5b799a16e8d05945d6dd993eac"} Oct 30 13:02:25.390056 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:25.390024 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" event={"ID":"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf","Type":"ContainerStarted","Data":"994775eeb5e92033429ace2d3d4fdf0d07b3edfa3e9be5b169f386c682316634"} Oct 30 13:02:26.034624 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:26.034587 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:26.036185 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:26.036163 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qvxgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 13:02:26.036258 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:26.036211 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" podUID="d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 13:02:27.034365 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:27.034327 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qvxgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 13:02:27.034751 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:27.034381 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" podUID="d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 13:02:28.033927 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:28.033888 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qvxgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 13:02:28.034164 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:28.033977 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" podUID="d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 13:02:29.034106 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:29.034071 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qvxgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 13:02:29.034493 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:29.034123 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" podUID="d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 13:02:29.700507 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:02:29.700464 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:02:30.034067 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:30.033979 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qvxgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 13:02:30.034067 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:30.034029 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" podUID="d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 13:02:31.034479 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:31.034435 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qvxgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 13:02:31.035044 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:31.034497 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" podUID="d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 13:02:32.625931 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:32.625887 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:32.626558 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:32.626291 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:32.627244 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:32.627225 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:02:32.688172 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:32.688116 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" podStartSLOduration=9.688099837 podStartE2EDuration="9.688099837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:25.48179789 +0000 UTC m=+639.305026167" watchObservedRunningTime="2026-10-30 13:02:32.688099837 +0000 UTC m=+646.511328114" Oct 30 13:02:48.863780 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:48.863747 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t"] Oct 30 13:02:48.872788 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:48.872762 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:48.892545 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:48.892518 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t"] Oct 30 13:02:48.901184 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:48.901152 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-ac0c58-openshift-default-dockercfg-5mbd5\"" Oct 30 13:02:48.983558 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:48.983516 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-workload-certs\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:48.983558 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:48.983560 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-token\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:48.983766 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:48.983652 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqff6\" (UniqueName: \"kubernetes.io/projected/52f2532a-6a21-4115-8b67-082b8b9ea83f-kube-api-access-nqff6\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:48.983766 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:48.983696 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-workload-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:48.983766 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:48.983757 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-podinfo\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:48.983868 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:48.983781 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-data\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:48.983868 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:48.983831 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-envoy\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:48.983963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:48.983879 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-credential-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:48.983963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:48.983912 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f2532a-6a21-4115-8b67-082b8b9ea83f-istiod-ca-cert\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.084709 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.084676 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-token\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.084902 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.084733 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqff6\" (UniqueName: \"kubernetes.io/projected/52f2532a-6a21-4115-8b67-082b8b9ea83f-kube-api-access-nqff6\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.084902 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.084758 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-workload-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.084902 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.084775 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-podinfo\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.084902 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.084793 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-data\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.084902 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.084817 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-envoy\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.085220 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.085163 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-credential-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.085279 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.085241 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f2532a-6a21-4115-8b67-082b8b9ea83f-istiod-ca-cert\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.085333 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.085270 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-workload-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.085333 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.085309 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-workload-certs\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.085566 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.085543 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-credential-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.085635 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.085615 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-workload-certs\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.085692 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.085678 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-data\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.085870 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.085850 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f2532a-6a21-4115-8b67-082b8b9ea83f-istiod-ca-cert\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.087804 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.087781 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-envoy\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.087987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.087971 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-podinfo\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.112035 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.112007 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-token\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.112216 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.112195 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqff6\" (UniqueName: \"kubernetes.io/projected/52f2532a-6a21-4115-8b67-082b8b9ea83f-kube-api-access-nqff6\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.146450 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.146417 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6"] Oct 30 13:02:49.152477 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.152450 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.174689 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.173774 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6"] Oct 30 13:02:49.186793 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.186760 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.186925 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.186845 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f0f890-bb6a-4627-9487-90741d34337c-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.186925 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.186880 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:49.187054 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.186886 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.187109 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.187080 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.187163 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.187132 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f0f890-bb6a-4627-9487-90741d34337c-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.187219 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.187199 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6tpm\" (UniqueName: \"kubernetes.io/projected/70f0f890-bb6a-4627-9487-90741d34337c-kube-api-access-x6tpm\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.187272 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.187242 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.187323 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.187275 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f0f890-bb6a-4627-9487-90741d34337c-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.187381 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.187326 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.215049 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.215021 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-42ee9d-openshift-default-dockercfg-sb4pn\"" Oct 30 13:02:49.288821 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.288793 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.289006 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.288858 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.289006 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.288895 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f0f890-bb6a-4627-9487-90741d34337c-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.289006 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.288929 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.289006 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.288977 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.289225 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.289017 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f0f890-bb6a-4627-9487-90741d34337c-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.289225 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.289077 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6tpm\" (UniqueName: \"kubernetes.io/projected/70f0f890-bb6a-4627-9487-90741d34337c-kube-api-access-x6tpm\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.289225 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.289114 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.289225 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.289145 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f0f890-bb6a-4627-9487-90741d34337c-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.290565 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.289806 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.290565 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.290036 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.290565 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.290299 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.290565 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.290446 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f0f890-bb6a-4627-9487-90741d34337c-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.290802 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.290679 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.292164 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.292144 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.293843 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.293824 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f0f890-bb6a-4627-9487-90741d34337c-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.318689 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.318665 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f0f890-bb6a-4627-9487-90741d34337c-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.319667 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.319628 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6tpm\" (UniqueName: \"kubernetes.io/projected/70f0f890-bb6a-4627-9487-90741d34337c-kube-api-access-x6tpm\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.472681 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.472589 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:49.516974 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.515854 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t"] Oct 30 13:02:49.523996 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:02:49.523965 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f2532a_6a21_4115_8b67_082b8b9ea83f.slice/crio-c60f52f5479b4a96b615c97ae9895b9e4fc3cadaf1bb7429fd2318a585e16e76 WatchSource:0}: Error finding container c60f52f5479b4a96b615c97ae9895b9e4fc3cadaf1bb7429fd2318a585e16e76: Status 404 returned error can't find the container with id c60f52f5479b4a96b615c97ae9895b9e4fc3cadaf1bb7429fd2318a585e16e76 Oct 30 13:02:49.526592 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.526556 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:49.526698 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.526641 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:49.526698 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.526691 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:49.573227 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.573196 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5"] Oct 30 13:02:49.579054 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.579032 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.601185 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.601160 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5"] Oct 30 13:02:49.605228 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.605206 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-54994c-openshift-default-dockercfg-2qhnp\"" Oct 30 13:02:49.692669 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.692166 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.692669 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.692249 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.692669 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.692295 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.692669 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.692322 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.692669 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.692354 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsjgv\" (UniqueName: \"kubernetes.io/projected/f263d119-9ab0-440f-ac86-fa5a0d2a2819-kube-api-access-rsjgv\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.692669 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.692382 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.692669 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.692425 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.692669 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.692471 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.692669 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.692498 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.794208 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.794128 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsjgv\" (UniqueName: \"kubernetes.io/projected/f263d119-9ab0-440f-ac86-fa5a0d2a2819-kube-api-access-rsjgv\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.794208 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.794181 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.794382 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.794235 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.794382 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.794290 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.794382 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.794319 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.794382 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.794365 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.794510 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.794462 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.794546 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.794519 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.794578 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.794545 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.795078 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.795028 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.795316 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.795271 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.795500 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.795334 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.795724 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.795706 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.796078 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.796035 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.797405 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.797386 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.798688 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.798668 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6"] Oct 30 13:02:49.799196 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.799173 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.802597 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.802568 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:49.802682 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.802632 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:49.802682 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.802659 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:49.828620 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.828554 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsjgv\" (UniqueName: \"kubernetes.io/projected/f263d119-9ab0-440f-ac86-fa5a0d2a2819-kube-api-access-rsjgv\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.831705 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.831680 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:49.915338 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:49.915102 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:50.224815 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:02:50.224774 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf263d119_9ab0_440f_ac86_fa5a0d2a2819.slice/crio-db2ae9cbd9d9ce17de76366f5a73d23f1f4eefb4ca76c12dee3c9bf493c66b68 WatchSource:0}: Error finding container db2ae9cbd9d9ce17de76366f5a73d23f1f4eefb4ca76c12dee3c9bf493c66b68: Status 404 returned error can't find the container with id db2ae9cbd9d9ce17de76366f5a73d23f1f4eefb4ca76c12dee3c9bf493c66b68 Oct 30 13:02:50.226092 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:50.226068 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5"] Oct 30 13:02:50.227279 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:50.227246 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:50.227360 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:50.227308 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:50.227360 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:50.227341 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:50.535981 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:50.535865 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" event={"ID":"52f2532a-6a21-4115-8b67-082b8b9ea83f","Type":"ContainerStarted","Data":"ce2add05a4f83079d26a9215f45be73d03e570c33bb4a5642dad2e0df1252656"} Oct 30 13:02:50.537392 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:50.537360 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" event={"ID":"52f2532a-6a21-4115-8b67-082b8b9ea83f","Type":"ContainerStarted","Data":"c60f52f5479b4a96b615c97ae9895b9e4fc3cadaf1bb7429fd2318a585e16e76"} Oct 30 13:02:50.538461 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:50.537846 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" event={"ID":"f263d119-9ab0-440f-ac86-fa5a0d2a2819","Type":"ContainerStarted","Data":"2912df6ca2092269d03f0287949e2156a18812e350585f9fbd2f9014230158c6"} Oct 30 13:02:50.538461 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:50.537895 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" event={"ID":"f263d119-9ab0-440f-ac86-fa5a0d2a2819","Type":"ContainerStarted","Data":"db2ae9cbd9d9ce17de76366f5a73d23f1f4eefb4ca76c12dee3c9bf493c66b68"} Oct 30 13:02:50.540785 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:50.540754 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" event={"ID":"70f0f890-bb6a-4627-9487-90741d34337c","Type":"ContainerStarted","Data":"509359a3e740377c45d71cd18c332ef3fc4ee611c47e789912fb7006ab378970"} Oct 30 13:02:50.540909 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:50.540798 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" event={"ID":"70f0f890-bb6a-4627-9487-90741d34337c","Type":"ContainerStarted","Data":"b765045eed14df8e8f5fc118b0e7e1a08a18eec691ca6bb6956a86681e6baa4b"} Oct 30 13:02:50.583394 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:50.583338 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" podStartSLOduration=2.58332369 podStartE2EDuration="2.58332369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:50.582693558 +0000 UTC m=+664.405921869" watchObservedRunningTime="2026-10-30 13:02:50.58332369 +0000 UTC m=+664.406551966" Oct 30 13:02:50.630517 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:50.630442 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" podStartSLOduration=1.630423414 podStartE2EDuration="1.630423414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:50.628065035 +0000 UTC m=+664.451293324" watchObservedRunningTime="2026-10-30 13:02:50.630423414 +0000 UTC m=+664.453651691" Oct 30 13:02:50.678315 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:50.678266 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" podStartSLOduration=1.678250018 podStartE2EDuration="1.678250018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:50.67648538 +0000 UTC m=+664.499713657" watchObservedRunningTime="2026-10-30 13:02:50.678250018 +0000 UTC m=+664.501478294" Oct 30 13:02:50.915961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:50.915902 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:51.187522 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:51.187422 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:51.192372 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:51.192344 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:51.473293 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:51.473206 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:51.478496 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:51.478465 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:51.546237 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:51.546170 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:51.546237 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:51.546243 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:51.546918 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:51.546890 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:02:51.547104 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:51.547086 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:02:51.921618 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:51.921589 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:52.551229 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:52.551195 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:52.552110 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:52.552091 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:02:55.211409 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:55.209691 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k"] Oct 30 13:02:55.222035 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:55.221889 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" Oct 30 13:02:55.239600 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:55.236245 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k"] Oct 30 13:02:55.252625 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:55.252591 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8rvn\" (UniqueName: \"kubernetes.io/projected/34fa7386-a0cb-4313-86fc-8a82c931598a-kube-api-access-z8rvn\") pod \"payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k\" (UID: \"34fa7386-a0cb-4313-86fc-8a82c931598a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" Oct 30 13:02:55.252752 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:55.252658 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34fa7386-a0cb-4313-86fc-8a82c931598a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k\" (UID: \"34fa7386-a0cb-4313-86fc-8a82c931598a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" Oct 30 13:02:55.253316 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:55.253285 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-3048f9\"" Oct 30 13:02:55.253432 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:55.253351 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-3048f9-dockercfg-szzbs\"" Oct 30 13:02:55.353824 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:55.353785 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8rvn\" (UniqueName: \"kubernetes.io/projected/34fa7386-a0cb-4313-86fc-8a82c931598a-kube-api-access-z8rvn\") pod \"payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k\" (UID: \"34fa7386-a0cb-4313-86fc-8a82c931598a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" Oct 30 13:02:55.354049 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:55.353836 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34fa7386-a0cb-4313-86fc-8a82c931598a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k\" (UID: \"34fa7386-a0cb-4313-86fc-8a82c931598a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" Oct 30 13:02:55.354599 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:55.354577 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34fa7386-a0cb-4313-86fc-8a82c931598a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k\" (UID: \"34fa7386-a0cb-4313-86fc-8a82c931598a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" Oct 30 13:02:55.378376 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:55.378344 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8rvn\" (UniqueName: \"kubernetes.io/projected/34fa7386-a0cb-4313-86fc-8a82c931598a-kube-api-access-z8rvn\") pod \"payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k\" (UID: \"34fa7386-a0cb-4313-86fc-8a82c931598a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" Oct 30 13:02:55.559309 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:55.559215 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" Oct 30 13:02:55.870731 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:55.870669 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k"] Oct 30 13:02:55.871612 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:02:55.871583 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34fa7386_a0cb_4313_86fc_8a82c931598a.slice/crio-45fdb0782c5a04cead7773e9b668126ebe2779a689a94992266e218804433081 WatchSource:0}: Error finding container 45fdb0782c5a04cead7773e9b668126ebe2779a689a94992266e218804433081: Status 404 returned error can't find the container with id 45fdb0782c5a04cead7773e9b668126ebe2779a689a94992266e218804433081 Oct 30 13:02:56.578626 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:56.578588 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" event={"ID":"34fa7386-a0cb-4313-86fc-8a82c931598a","Type":"ContainerStarted","Data":"45fdb0782c5a04cead7773e9b668126ebe2779a689a94992266e218804433081"} Oct 30 13:02:57.039485 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:57.037018 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr"] Oct 30 13:02:57.053148 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:57.053102 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr"] Oct 30 13:02:57.053319 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:57.053230 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" Oct 30 13:02:57.080321 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:57.080293 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-bb71e2-dockercfg-xxlbt\"" Oct 30 13:02:57.080629 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:57.080590 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-bb71e2\"" Oct 30 13:02:57.084880 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:57.084852 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vspbp\" (UniqueName: \"kubernetes.io/projected/9402ca3b-a5e0-49b8-83a2-e84331209e84-kube-api-access-vspbp\") pod \"payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr\" (UID: \"9402ca3b-a5e0-49b8-83a2-e84331209e84\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" Oct 30 13:02:57.085011 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:57.084983 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9402ca3b-a5e0-49b8-83a2-e84331209e84-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr\" (UID: \"9402ca3b-a5e0-49b8-83a2-e84331209e84\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" Oct 30 13:02:57.186994 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:57.185995 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vspbp\" (UniqueName: \"kubernetes.io/projected/9402ca3b-a5e0-49b8-83a2-e84331209e84-kube-api-access-vspbp\") pod \"payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr\" (UID: \"9402ca3b-a5e0-49b8-83a2-e84331209e84\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" Oct 30 13:02:57.186994 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:57.186100 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9402ca3b-a5e0-49b8-83a2-e84331209e84-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr\" (UID: \"9402ca3b-a5e0-49b8-83a2-e84331209e84\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" Oct 30 13:02:57.186994 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:57.186973 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9402ca3b-a5e0-49b8-83a2-e84331209e84-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr\" (UID: \"9402ca3b-a5e0-49b8-83a2-e84331209e84\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" Oct 30 13:02:57.211852 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:57.211815 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vspbp\" (UniqueName: \"kubernetes.io/projected/9402ca3b-a5e0-49b8-83a2-e84331209e84-kube-api-access-vspbp\") pod \"payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr\" (UID: \"9402ca3b-a5e0-49b8-83a2-e84331209e84\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" Oct 30 13:02:57.375027 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:57.374207 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" Oct 30 13:02:57.697986 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:57.697500 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr"] Oct 30 13:02:58.600011 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:58.599969 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" event={"ID":"9402ca3b-a5e0-49b8-83a2-e84331209e84","Type":"ContainerStarted","Data":"5191541214c3fe1989d8ba4d2e614e17da8d8f3e5a411cffcaec61d9fdebce93"} Oct 30 13:02:58.931123 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:58.930788 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr"] Oct 30 13:02:58.959393 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:58.958126 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr"] Oct 30 13:02:58.959393 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:58.958380 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" Oct 30 13:02:58.989292 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:58.989247 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-175831\"" Oct 30 13:02:59.000800 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:59.000772 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-175831-dockercfg-xv7qs\"" Oct 30 13:02:59.022909 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:59.022874 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e386dc-1501-492d-a339-f83aa21f24fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr\" (UID: \"d1e386dc-1501-492d-a339-f83aa21f24fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" Oct 30 13:02:59.023073 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:59.023046 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l46x5\" (UniqueName: \"kubernetes.io/projected/d1e386dc-1501-492d-a339-f83aa21f24fe-kube-api-access-l46x5\") pod \"payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr\" (UID: \"d1e386dc-1501-492d-a339-f83aa21f24fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" Oct 30 13:02:59.123860 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:59.123828 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e386dc-1501-492d-a339-f83aa21f24fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr\" (UID: \"d1e386dc-1501-492d-a339-f83aa21f24fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" Oct 30 13:02:59.124025 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:59.123953 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l46x5\" (UniqueName: \"kubernetes.io/projected/d1e386dc-1501-492d-a339-f83aa21f24fe-kube-api-access-l46x5\") pod \"payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr\" (UID: \"d1e386dc-1501-492d-a339-f83aa21f24fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" Oct 30 13:02:59.124765 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:59.124701 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e386dc-1501-492d-a339-f83aa21f24fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr\" (UID: \"d1e386dc-1501-492d-a339-f83aa21f24fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" Oct 30 13:02:59.148810 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:59.148781 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l46x5\" (UniqueName: \"kubernetes.io/projected/d1e386dc-1501-492d-a339-f83aa21f24fe-kube-api-access-l46x5\") pod \"payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr\" (UID: \"d1e386dc-1501-492d-a339-f83aa21f24fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" Oct 30 13:02:59.291641 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:59.291608 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" Oct 30 13:02:59.590784 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:59.590757 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr"] Oct 30 13:02:59.592352 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:02:59.592317 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e386dc_1501_492d_a339_f83aa21f24fe.slice/crio-0b4098b175cdca1dbfcb1665ec4cffc0fb24015191da6820a0a58eeb8010b3b7 WatchSource:0}: Error finding container 0b4098b175cdca1dbfcb1665ec4cffc0fb24015191da6820a0a58eeb8010b3b7: Status 404 returned error can't find the container with id 0b4098b175cdca1dbfcb1665ec4cffc0fb24015191da6820a0a58eeb8010b3b7 Oct 30 13:02:59.607114 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:59.607080 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" event={"ID":"34fa7386-a0cb-4313-86fc-8a82c931598a","Type":"ContainerStarted","Data":"b768bdbc0e3bf9b47cbfbbc68858f4dc7228102f4db076ef8ce6bdd1b925252d"} Oct 30 13:02:59.607249 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:59.607188 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" Oct 30 13:02:59.608626 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:59.608595 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" event={"ID":"d1e386dc-1501-492d-a339-f83aa21f24fe","Type":"ContainerStarted","Data":"0b4098b175cdca1dbfcb1665ec4cffc0fb24015191da6820a0a58eeb8010b3b7"} Oct 30 13:02:59.610260 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:59.610235 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" event={"ID":"9402ca3b-a5e0-49b8-83a2-e84331209e84","Type":"ContainerStarted","Data":"32a5b2ed648298ecd1152492b95b401d0c3000ab4898cd45813534e06acb98ec"} Oct 30 13:02:59.610401 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:59.610383 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" Oct 30 13:02:59.652103 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:59.652043 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" podStartSLOduration=2.007493201 podStartE2EDuration="4.652023981s" totalImagesPullingTime="2.64453078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:59.649395937 +0000 UTC m=+673.472624213" watchObservedRunningTime="2026-10-30 13:02:59.652023981 +0000 UTC m=+673.475252259" Oct 30 13:02:59.691339 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:02:59.691288 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" podStartSLOduration=2.277395407 podStartE2EDuration="3.691273077s" totalImagesPullingTime="1.41387767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:59.690115831 +0000 UTC m=+673.513344109" watchObservedRunningTime="2026-10-30 13:02:59.691273077 +0000 UTC m=+673.514501418" Oct 30 13:02:59.704914 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:02:59.704852 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:03:00.618324 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:00.618285 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" event={"ID":"d1e386dc-1501-492d-a339-f83aa21f24fe","Type":"ContainerStarted","Data":"14bc85bc358c2ebc6d541ffbd34292dff64c4a2d8241629defbb97468d2efd26"} Oct 30 13:03:00.618850 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:00.618685 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" Oct 30 13:03:00.663491 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:00.663445 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" podStartSLOduration=2.663426393 podStartE2EDuration="2.663426393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:00.661154072 +0000 UTC m=+674.484382370" watchObservedRunningTime="2026-10-30 13:03:00.663426393 +0000 UTC m=+674.486654669" Oct 30 13:03:01.756497 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:01.756460 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv"] Oct 30 13:03:01.762246 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:01.762213 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" Oct 30 13:03:01.781579 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:01.781546 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv"] Oct 30 13:03:01.789799 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:01.789773 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-42ee9d-dockercfg-vm8lf\"" Oct 30 13:03:01.790056 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:01.789853 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-42ee9d\"" Oct 30 13:03:01.864752 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:01.864707 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803ac7a6-439b-4634-a116-5e16c2595f42-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv\" (UID: \"803ac7a6-439b-4634-a116-5e16c2595f42\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" Oct 30 13:03:01.864752 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:01.864766 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh65k\" (UniqueName: \"kubernetes.io/projected/803ac7a6-439b-4634-a116-5e16c2595f42-kube-api-access-vh65k\") pod \"payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv\" (UID: \"803ac7a6-439b-4634-a116-5e16c2595f42\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" Oct 30 13:03:01.965636 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:01.965591 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803ac7a6-439b-4634-a116-5e16c2595f42-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv\" (UID: \"803ac7a6-439b-4634-a116-5e16c2595f42\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" Oct 30 13:03:01.965837 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:01.965647 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh65k\" (UniqueName: \"kubernetes.io/projected/803ac7a6-439b-4634-a116-5e16c2595f42-kube-api-access-vh65k\") pod \"payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv\" (UID: \"803ac7a6-439b-4634-a116-5e16c2595f42\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" Oct 30 13:03:01.966598 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:01.966567 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803ac7a6-439b-4634-a116-5e16c2595f42-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv\" (UID: \"803ac7a6-439b-4634-a116-5e16c2595f42\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" Oct 30 13:03:02.008378 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.005696 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh65k\" (UniqueName: \"kubernetes.io/projected/803ac7a6-439b-4634-a116-5e16c2595f42-kube-api-access-vh65k\") pod \"payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv\" (UID: \"803ac7a6-439b-4634-a116-5e16c2595f42\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" Oct 30 13:03:02.078223 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.078189 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" Oct 30 13:03:02.399726 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:03:02.399689 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803ac7a6_439b_4634_a116_5e16c2595f42.slice/crio-f3c9daae029960745ea1a5f661bb15394bb1c0f1fb66e38bd9db11a5c9620b88 WatchSource:0}: Error finding container f3c9daae029960745ea1a5f661bb15394bb1c0f1fb66e38bd9db11a5c9620b88: Status 404 returned error can't find the container with id f3c9daae029960745ea1a5f661bb15394bb1c0f1fb66e38bd9db11a5c9620b88 Oct 30 13:03:02.400220 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.400181 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv"] Oct 30 13:03:02.448915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.448888 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q"] Oct 30 13:03:02.454196 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.454173 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" Oct 30 13:03:02.485353 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.480589 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-175831-serving-cert\"" Oct 30 13:03:02.485353 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.482564 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d5n24\"" Oct 30 13:03:02.485353 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.484795 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 13:03:02.485353 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.485281 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q"] Oct 30 13:03:02.583630 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.583591 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a70b19-c9b4-436f-9a48-a45fd8d158e0-metrics-certs\") pod \"maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q\" (UID: \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" Oct 30 13:03:02.583834 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.583654 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a70b19-c9b4-436f-9a48-a45fd8d158e0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q\" (UID: \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" Oct 30 13:03:02.583834 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.583799 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a70b19-c9b4-436f-9a48-a45fd8d158e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q\" (UID: \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" Oct 30 13:03:02.584037 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.583848 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdr84\" (UniqueName: \"kubernetes.io/projected/15a70b19-c9b4-436f-9a48-a45fd8d158e0-kube-api-access-cdr84\") pod \"maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q\" (UID: \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" Oct 30 13:03:02.632669 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.632630 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" event={"ID":"803ac7a6-439b-4634-a116-5e16c2595f42","Type":"ContainerStarted","Data":"6867ab281b6bf64d6dfb629e3c907371b7390cad875cc8e85e44ca3427e9afac"} Oct 30 13:03:02.632843 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.632688 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" event={"ID":"803ac7a6-439b-4634-a116-5e16c2595f42","Type":"ContainerStarted","Data":"f3c9daae029960745ea1a5f661bb15394bb1c0f1fb66e38bd9db11a5c9620b88"} Oct 30 13:03:02.632843 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.632710 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" Oct 30 13:03:02.673555 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.673450 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" podStartSLOduration=1.6734323880000002 podStartE2EDuration="1.673432388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:02.673308597 +0000 UTC m=+676.496536874" watchObservedRunningTime="2026-10-30 13:03:02.673432388 +0000 UTC m=+676.496660666" Oct 30 13:03:02.684822 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.684788 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a70b19-c9b4-436f-9a48-a45fd8d158e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q\" (UID: \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" Oct 30 13:03:02.684977 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.684845 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdr84\" (UniqueName: \"kubernetes.io/projected/15a70b19-c9b4-436f-9a48-a45fd8d158e0-kube-api-access-cdr84\") pod \"maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q\" (UID: \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" Oct 30 13:03:02.684977 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.684874 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a70b19-c9b4-436f-9a48-a45fd8d158e0-metrics-certs\") pod \"maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q\" (UID: \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" Oct 30 13:03:02.684977 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.684916 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a70b19-c9b4-436f-9a48-a45fd8d158e0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q\" (UID: \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" Oct 30 13:03:02.685655 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.685629 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a70b19-c9b4-436f-9a48-a45fd8d158e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q\" (UID: \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" Oct 30 13:03:02.687497 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.687470 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a70b19-c9b4-436f-9a48-a45fd8d158e0-metrics-certs\") pod \"maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q\" (UID: \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" Oct 30 13:03:02.687673 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.687654 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a70b19-c9b4-436f-9a48-a45fd8d158e0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q\" (UID: \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" Oct 30 13:03:02.709396 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.709366 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdr84\" (UniqueName: \"kubernetes.io/projected/15a70b19-c9b4-436f-9a48-a45fd8d158e0-kube-api-access-cdr84\") pod \"maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q\" (UID: \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" Oct 30 13:03:02.769460 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:02.769421 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" Oct 30 13:03:03.101155 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:03:03.101113 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a70b19_c9b4_436f_9a48_a45fd8d158e0.slice/crio-b7de9d06b5e1fb79e1089c9adc925a49d64d4fe59505b3ffd195650664ab7f33 WatchSource:0}: Error finding container b7de9d06b5e1fb79e1089c9adc925a49d64d4fe59505b3ffd195650664ab7f33: Status 404 returned error can't find the container with id b7de9d06b5e1fb79e1089c9adc925a49d64d4fe59505b3ffd195650664ab7f33 Oct 30 13:03:03.102879 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:03.102857 4503 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:03:03.103060 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:03.103033 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q"] Oct 30 13:03:03.641746 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:03.641703 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" event={"ID":"15a70b19-c9b4-436f-9a48-a45fd8d158e0","Type":"ContainerStarted","Data":"b7de9d06b5e1fb79e1089c9adc925a49d64d4fe59505b3ffd195650664ab7f33"} Oct 30 13:03:03.727599 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:03.727561 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw"] Oct 30 13:03:03.731850 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:03.731822 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw" Oct 30 13:03:03.748246 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:03.748193 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-ac0c58\"" Oct 30 13:03:03.748246 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:03.748231 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-ac0c58-dockercfg-cgdbx\"" Oct 30 13:03:03.757149 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:03.757118 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw"] Oct 30 13:03:03.801243 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:03.801210 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sr4j\" (UniqueName: \"kubernetes.io/projected/738b8f50-96a6-4a49-b5f7-92b1b9fb5975-kube-api-access-7sr4j\") pod \"payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw\" (UID: \"738b8f50-96a6-4a49-b5f7-92b1b9fb5975\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw" Oct 30 13:03:03.801660 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:03.801359 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/738b8f50-96a6-4a49-b5f7-92b1b9fb5975-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw\" (UID: \"738b8f50-96a6-4a49-b5f7-92b1b9fb5975\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw" Oct 30 13:03:03.906961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:03.905589 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sr4j\" (UniqueName: \"kubernetes.io/projected/738b8f50-96a6-4a49-b5f7-92b1b9fb5975-kube-api-access-7sr4j\") pod \"payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw\" (UID: \"738b8f50-96a6-4a49-b5f7-92b1b9fb5975\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw" Oct 30 13:03:03.906961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:03.905704 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/738b8f50-96a6-4a49-b5f7-92b1b9fb5975-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw\" (UID: \"738b8f50-96a6-4a49-b5f7-92b1b9fb5975\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw" Oct 30 13:03:03.906961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:03.906668 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/738b8f50-96a6-4a49-b5f7-92b1b9fb5975-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw\" (UID: \"738b8f50-96a6-4a49-b5f7-92b1b9fb5975\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw" Oct 30 13:03:03.935918 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:03.935885 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sr4j\" (UniqueName: \"kubernetes.io/projected/738b8f50-96a6-4a49-b5f7-92b1b9fb5975-kube-api-access-7sr4j\") pod \"payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw\" (UID: \"738b8f50-96a6-4a49-b5f7-92b1b9fb5975\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw" Oct 30 13:03:04.049162 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:04.049123 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw" Oct 30 13:03:04.369700 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:03:04.369427 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738b8f50_96a6_4a49_b5f7_92b1b9fb5975.slice/crio-a27a73680138853938cc3ba617cc8eb923c95b5ef99677f2385326b3ec61d93a WatchSource:0}: Error finding container a27a73680138853938cc3ba617cc8eb923c95b5ef99677f2385326b3ec61d93a: Status 404 returned error can't find the container with id a27a73680138853938cc3ba617cc8eb923c95b5ef99677f2385326b3ec61d93a Oct 30 13:03:04.369700 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:04.369639 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw"] Oct 30 13:03:04.650749 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:04.650710 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw" event={"ID":"738b8f50-96a6-4a49-b5f7-92b1b9fb5975","Type":"ContainerStarted","Data":"7015fa22b21a65f0cea24e6026ae78af2c4e197d1074f5db4119e17307818502"} Oct 30 13:03:04.651020 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:04.650776 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw" event={"ID":"738b8f50-96a6-4a49-b5f7-92b1b9fb5975","Type":"ContainerStarted","Data":"a27a73680138853938cc3ba617cc8eb923c95b5ef99677f2385326b3ec61d93a"} Oct 30 13:03:04.651679 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:04.651655 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw" Oct 30 13:03:04.694691 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:04.694639 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw" podStartSLOduration=1.6946247159999999 podStartE2EDuration="1.694624716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:04.692515956 +0000 UTC m=+678.515744248" watchObservedRunningTime="2026-10-30 13:03:04.694624716 +0000 UTC m=+678.517853046" Oct 30 13:03:05.690990 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:05.690928 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" event={"ID":"15a70b19-c9b4-436f-9a48-a45fd8d158e0","Type":"ContainerStarted","Data":"1cf6e46021d8a5fee5758048e06390a58bddabab7195f0bb38f3ea8f731ac5a1"} Oct 30 13:03:05.691499 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:05.691043 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" Oct 30 13:03:05.735406 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:05.735349 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" podStartSLOduration=2.097115887 podStartE2EDuration="3.73533541s" totalImagesPullingTime="1.638219523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:05.734162662 +0000 UTC m=+679.557390939" watchObservedRunningTime="2026-10-30 13:03:05.73533541 +0000 UTC m=+679.558563687" Oct 30 13:03:08.039715 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:08.039678 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf"] Oct 30 13:03:08.043759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:08.043730 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" Oct 30 13:03:08.074354 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:08.074314 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-ac0c58-serving-cert\"" Oct 30 13:03:08.077559 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:08.077528 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf"] Oct 30 13:03:08.155529 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:08.155491 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-metrics-certs\") pod \"maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf\" (UID: \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" Oct 30 13:03:08.155692 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:08.155657 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf\" (UID: \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" Oct 30 13:03:08.155776 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:08.155704 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjfkw\" (UniqueName: \"kubernetes.io/projected/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-kube-api-access-bjfkw\") pod \"maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf\" (UID: \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" Oct 30 13:03:08.155829 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:08.155778 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-maas-api-tls\") pod \"maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf\" (UID: \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" Oct 30 13:03:08.257032 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:08.256992 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-metrics-certs\") pod \"maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf\" (UID: \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" Oct 30 13:03:08.257205 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:08.257095 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf\" (UID: \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" Oct 30 13:03:08.257205 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:08.257139 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjfkw\" (UniqueName: \"kubernetes.io/projected/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-kube-api-access-bjfkw\") pod \"maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf\" (UID: \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" Oct 30 13:03:08.257205 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:08.257166 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-maas-api-tls\") pod \"maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf\" (UID: \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" Oct 30 13:03:08.257381 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:03:08.257293 4503 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-serving-cert: secret "maas-api-e2e-worker-w1-ac0c58-serving-cert" not found Oct 30 13:03:08.257381 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:03:08.257366 4503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-maas-api-tls podName:c3c12544-6e6c-487a-ad99-bb5d0f5da8aa nodeName:}" failed. No retries permitted until 2026-10-30 13:03:08.757345325 +0000 UTC m=+682.580573583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-maas-api-tls") pod "maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" (UID: "c3c12544-6e6c-487a-ad99-bb5d0f5da8aa") : secret "maas-api-e2e-worker-w1-ac0c58-serving-cert" not found Oct 30 13:03:08.257846 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:08.257825 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf\" (UID: \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" Oct 30 13:03:08.259537 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:08.259515 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-metrics-certs\") pod \"maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf\" (UID: \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" Oct 30 13:03:08.284626 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:08.284588 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjfkw\" (UniqueName: \"kubernetes.io/projected/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-kube-api-access-bjfkw\") pod \"maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf\" (UID: \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" Oct 30 13:03:08.762670 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:08.762626 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-maas-api-tls\") pod \"maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf\" (UID: \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" Oct 30 13:03:08.765383 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:08.765350 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-maas-api-tls\") pod \"maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf\" (UID: \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" Oct 30 13:03:08.956925 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:08.956871 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" Oct 30 13:03:09.282146 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:03:09.282107 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c12544_6e6c_487a_ad99_bb5d0f5da8aa.slice/crio-836403b9ae41d49be179c10e1d513c52b98c9c5f48033ffee81ca144b31b73e6 WatchSource:0}: Error finding container 836403b9ae41d49be179c10e1d513c52b98c9c5f48033ffee81ca144b31b73e6: Status 404 returned error can't find the container with id 836403b9ae41d49be179c10e1d513c52b98c9c5f48033ffee81ca144b31b73e6 Oct 30 13:03:09.282570 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:09.282206 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf"] Oct 30 13:03:09.715340 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:09.715306 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" event={"ID":"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa","Type":"ContainerStarted","Data":"836403b9ae41d49be179c10e1d513c52b98c9c5f48033ffee81ca144b31b73e6"} Oct 30 13:03:10.621756 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:10.621727 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" Oct 30 13:03:10.622142 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:10.621792 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" Oct 30 13:03:10.724585 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:10.724543 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" event={"ID":"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa","Type":"ContainerStarted","Data":"5831ec2f76cd31572b8db0c77700db7d44bc466fdecf2d7a4ba9985173459059"} Oct 30 13:03:10.724751 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:10.724732 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" Oct 30 13:03:10.804662 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:10.804594 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" podStartSLOduration=2.409564298 podStartE2EDuration="2.804573843s" totalImagesPullingTime="395.009545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:10.788655123 +0000 UTC m=+684.611883401" watchObservedRunningTime="2026-10-30 13:03:10.804573843 +0000 UTC m=+684.627802140" Oct 30 13:03:11.628309 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:11.628281 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" Oct 30 13:03:11.714537 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:11.714506 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" Oct 30 13:03:13.664965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:13.664116 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" Oct 30 13:03:16.702107 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:16.702004 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw" Oct 30 13:03:16.737299 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:16.737267 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" Oct 30 13:03:20.953964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:20.953279 4503 generic.go:356] "Generic (PLEG): container finished" podID="f263d119-9ab0-440f-ac86-fa5a0d2a2819" containerID="2912df6ca2092269d03f0287949e2156a18812e350585f9fbd2f9014230158c6" exitCode=0 Oct 30 13:03:20.953964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:20.953524 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" event={"ID":"f263d119-9ab0-440f-ac86-fa5a0d2a2819","Type":"ContainerDied","Data":"2912df6ca2092269d03f0287949e2156a18812e350585f9fbd2f9014230158c6"} Oct 30 13:03:20.955569 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:20.955525 4503 scope.go:122] "RemoveContainer" containerID="2912df6ca2092269d03f0287949e2156a18812e350585f9fbd2f9014230158c6" Oct 30 13:03:20.958349 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:20.958283 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:20.958503 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:20.958406 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:20.958503 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:20.958456 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:21.570018 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:21.567094 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 13:03:21.570018 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:21.567164 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" podUID="70f0f890-bb6a-4627-9487-90741d34337c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 13:03:21.978712 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:21.978675 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" event={"ID":"f263d119-9ab0-440f-ac86-fa5a0d2a2819","Type":"ContainerStarted","Data":"ef7a4c1168ffbbdd707284db9cfa28b3d20fd3743f7d9e6bd856341ff435e7e4"} Oct 30 13:03:21.980504 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:21.980456 4503 generic.go:356] "Generic (PLEG): container finished" podID="70f0f890-bb6a-4627-9487-90741d34337c" containerID="509359a3e740377c45d71cd18c332ef3fc4ee611c47e789912fb7006ab378970" exitCode=0 Oct 30 13:03:21.980638 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:21.980538 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" event={"ID":"70f0f890-bb6a-4627-9487-90741d34337c","Type":"ContainerDied","Data":"509359a3e740377c45d71cd18c332ef3fc4ee611c47e789912fb7006ab378970"} Oct 30 13:03:21.980887 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:21.980872 4503 scope.go:122] "RemoveContainer" containerID="509359a3e740377c45d71cd18c332ef3fc4ee611c47e789912fb7006ab378970" Oct 30 13:03:21.982183 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:21.982147 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:21.982276 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:21.982223 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:21.982276 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:21.982255 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:22.551217 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:22.551180 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:03:22.593477 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:22.593132 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:03:22.593477 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:22.593199 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" podUID="52f2532a-6a21-4115-8b67-082b8b9ea83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:22.915378 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:22.915347 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:03:22.916540 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:22.916507 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 13:03:22.916651 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:22.916587 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" podUID="f263d119-9ab0-440f-ac86-fa5a0d2a2819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 13:03:23.022875 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:23.020495 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" event={"ID":"70f0f890-bb6a-4627-9487-90741d34337c","Type":"ContainerStarted","Data":"d50382081c56a2f580490f8b8cbd3a4025d957d1c04889923a3389a2cc6b4573"} Oct 30 13:03:23.473432 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:23.473392 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:03:23.473796 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:23.473774 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 13:03:23.473977 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:23.473924 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" podUID="70f0f890-bb6a-4627-9487-90741d34337c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 13:03:23.915821 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:23.915459 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 13:03:23.915821 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:23.915509 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" podUID="f263d119-9ab0-440f-ac86-fa5a0d2a2819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 13:03:24.474369 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:24.474331 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 13:03:24.475001 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:24.474966 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" podUID="70f0f890-bb6a-4627-9487-90741d34337c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 13:03:24.916472 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:24.916443 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 13:03:24.916630 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:24.916502 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" podUID="f263d119-9ab0-440f-ac86-fa5a0d2a2819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 13:03:25.479965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:25.475834 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 13:03:25.479965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:25.475896 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" podUID="70f0f890-bb6a-4627-9487-90741d34337c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 13:03:25.928022 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:25.916577 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 13:03:25.928022 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:25.916633 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" podUID="f263d119-9ab0-440f-ac86-fa5a0d2a2819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 13:03:26.476769 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:26.476315 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 13:03:26.476769 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:26.476377 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" podUID="70f0f890-bb6a-4627-9487-90741d34337c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 13:03:26.915661 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:26.915618 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 13:03:26.916155 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:26.915677 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" podUID="f263d119-9ab0-440f-ac86-fa5a0d2a2819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 13:03:27.473475 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:27.473434 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 13:03:27.473771 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:27.473738 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" podUID="70f0f890-bb6a-4627-9487-90741d34337c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 13:03:27.920261 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:27.920229 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:03:27.921186 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:27.921165 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:03:28.084398 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:28.084276 4503 generic.go:356] "Generic (PLEG): container finished" podID="52f2532a-6a21-4115-8b67-082b8b9ea83f" containerID="ce2add05a4f83079d26a9215f45be73d03e570c33bb4a5642dad2e0df1252656" exitCode=0 Oct 30 13:03:28.085992 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:28.085060 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" event={"ID":"52f2532a-6a21-4115-8b67-082b8b9ea83f","Type":"ContainerDied","Data":"ce2add05a4f83079d26a9215f45be73d03e570c33bb4a5642dad2e0df1252656"} Oct 30 13:03:28.085992 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:28.085390 4503 scope.go:122] "RemoveContainer" containerID="ce2add05a4f83079d26a9215f45be73d03e570c33bb4a5642dad2e0df1252656" Oct 30 13:03:28.086879 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:28.086847 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:28.087002 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:28.086925 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:28.087002 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:28.086976 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:03:28.767391 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:28.767358 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:03:28.769166 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:28.769141 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:03:28.774173 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:28.774150 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:03:29.092856 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:29.092776 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" event={"ID":"52f2532a-6a21-4115-8b67-082b8b9ea83f","Type":"ContainerStarted","Data":"d431d4727b5ac6eb489f50ff7641f585b67b64ccb7ab7f45ad1fc8e5c0be2a31"} Oct 30 13:03:29.187260 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:29.187224 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:03:29.187448 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:29.187396 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 13:03:29.187522 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:29.187448 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" podUID="52f2532a-6a21-4115-8b67-082b8b9ea83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 13:03:29.703201 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:03:29.703150 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:03:30.187632 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:30.187601 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 13:03:30.188053 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:30.187651 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" podUID="52f2532a-6a21-4115-8b67-082b8b9ea83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 13:03:31.189632 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:31.189589 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 13:03:31.190188 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:31.189661 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" podUID="52f2532a-6a21-4115-8b67-082b8b9ea83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 13:03:32.187210 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:32.187176 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 13:03:32.187382 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:32.187259 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" podUID="52f2532a-6a21-4115-8b67-082b8b9ea83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 13:03:33.187167 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:33.187129 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 13:03:33.187704 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:33.187185 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" podUID="52f2532a-6a21-4115-8b67-082b8b9ea83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 13:03:34.188070 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:34.188037 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 13:03:34.188555 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:34.188094 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" podUID="52f2532a-6a21-4115-8b67-082b8b9ea83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 13:03:35.205865 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:35.205827 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:03:35.206903 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:35.206797 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:03:35.208294 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:03:35.208270 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:03:59.701699 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:03:59.701658 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:04:04.567854 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.567813 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6"] Oct 30 13:04:04.570717 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.570689 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.587194 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.587164 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3048f9-openshift-default-dockercfg-b5fdc\"" Oct 30 13:04:04.601634 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.601595 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6"] Oct 30 13:04:04.657336 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.657292 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-podinfo\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.657512 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.657458 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-envoy\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.657512 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.657502 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl84v\" (UniqueName: \"kubernetes.io/projected/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-kube-api-access-dl84v\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.657638 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.657614 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-workload-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.657679 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.657650 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-token\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.657723 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.657684 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-workload-certs\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.657759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.657732 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istiod-ca-cert\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.657795 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.657782 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-data\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.657839 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.657808 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-credential-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.759230 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.759189 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-workload-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.759230 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.759228 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-token\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.759485 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.759264 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-workload-certs\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.759485 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.759292 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istiod-ca-cert\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.759485 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.759327 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-data\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.759485 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.759416 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-credential-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.759718 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.759502 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-podinfo\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.759718 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.759622 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-envoy\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.759718 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.759664 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl84v\" (UniqueName: \"kubernetes.io/projected/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-kube-api-access-dl84v\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.759718 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.759706 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-data\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.759914 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.759726 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-workload-certs\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.759914 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.759800 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-workload-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.760007 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.759914 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-credential-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.760218 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.760192 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istiod-ca-cert\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.762135 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.762114 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-envoy\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.762243 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.762207 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-podinfo\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.784294 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.784267 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-token\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.784878 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.784840 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl84v\" (UniqueName: \"kubernetes.io/projected/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-kube-api-access-dl84v\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:04.883814 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:04.883720 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:05.212385 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:05.212354 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6"] Oct 30 13:04:05.214416 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:04:05.214382 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ebe375_c4db_41eb_bc87_3cb9d8e0853b.slice/crio-a022bab8d1b3991aa9d9ce49e201a57b8f4394906ce651045a2abbaa8a77e0f9 WatchSource:0}: Error finding container a022bab8d1b3991aa9d9ce49e201a57b8f4394906ce651045a2abbaa8a77e0f9: Status 404 returned error can't find the container with id a022bab8d1b3991aa9d9ce49e201a57b8f4394906ce651045a2abbaa8a77e0f9 Oct 30 13:04:05.216973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:05.216905 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:04:05.217077 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:05.217017 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:04:05.217077 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:05.217068 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:04:05.348960 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:05.348493 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" event={"ID":"25ebe375-c4db-41eb-bc87-3cb9d8e0853b","Type":"ContainerStarted","Data":"3db6d80a5e252a56b71a9a0666c47917154ee121516f7df5b9e48d6449aa798a"} Oct 30 13:04:05.348960 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:05.348555 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" event={"ID":"25ebe375-c4db-41eb-bc87-3cb9d8e0853b","Type":"ContainerStarted","Data":"a022bab8d1b3991aa9d9ce49e201a57b8f4394906ce651045a2abbaa8a77e0f9"} Oct 30 13:04:05.397545 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:05.397495 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" podStartSLOduration=1.397480773 podStartE2EDuration="1.397480773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:05.395007258 +0000 UTC m=+739.218235540" watchObservedRunningTime="2026-10-30 13:04:05.397480773 +0000 UTC m=+739.220709050" Oct 30 13:04:05.884039 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:05.883995 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:06.886115 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:06.886078 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 13:04:06.886512 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:06.886134 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" podUID="25ebe375-c4db-41eb-bc87-3cb9d8e0853b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 13:04:07.886960 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:07.886313 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 13:04:07.886960 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:07.886377 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" podUID="25ebe375-c4db-41eb-bc87-3cb9d8e0853b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 13:04:08.884646 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:08.884608 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 13:04:08.884823 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:08.884666 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" podUID="25ebe375-c4db-41eb-bc87-3cb9d8e0853b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 13:04:09.884837 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:09.884800 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 13:04:09.885268 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:09.884870 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" podUID="25ebe375-c4db-41eb-bc87-3cb9d8e0853b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 13:04:10.884766 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:10.884732 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 13:04:10.885038 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:10.885012 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" podUID="25ebe375-c4db-41eb-bc87-3cb9d8e0853b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 13:04:12.228378 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:12.228336 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:12.228869 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:12.228798 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:12.230239 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:12.230216 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:04:29.702120 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:04:29.702031 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:04:33.952672 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:33.952635 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h"] Oct 30 13:04:33.955705 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:33.955677 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:33.979082 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:33.979058 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-484607-openshift-default-dockercfg-ggcps\"" Oct 30 13:04:34.013677 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.013639 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h"] Oct 30 13:04:34.030680 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.030646 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.030852 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.030730 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.030905 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.030876 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.030988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.030930 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.031042 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.031027 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1423ef5-6c20-40a5-a870-4934b313ba4c-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.031115 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.031099 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.031150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.031132 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.031182 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.031151 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5kzh\" (UniqueName: \"kubernetes.io/projected/c1423ef5-6c20-40a5-a870-4934b313ba4c-kube-api-access-x5kzh\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.031261 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.031244 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.132229 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.132180 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.132423 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.132250 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.132423 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.132283 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.132423 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.132317 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1423ef5-6c20-40a5-a870-4934b313ba4c-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.132423 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.132370 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.132423 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.132407 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.132423 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.132424 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5kzh\" (UniqueName: \"kubernetes.io/projected/c1423ef5-6c20-40a5-a870-4934b313ba4c-kube-api-access-x5kzh\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.132744 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.132446 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.132744 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.132465 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.132845 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.132789 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.133018 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.132992 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.133471 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.133446 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.133567 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.133535 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.133629 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.133614 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1423ef5-6c20-40a5-a870-4934b313ba4c-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.135164 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.135138 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.135530 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.135511 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.157490 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.157460 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.157958 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.157921 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5kzh\" (UniqueName: \"kubernetes.io/projected/c1423ef5-6c20-40a5-a870-4934b313ba4c-kube-api-access-x5kzh\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.272462 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.272371 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:34.609290 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.609260 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h"] Oct 30 13:04:34.611658 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:04:34.611631 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1423ef5_6c20_40a5_a870_4934b313ba4c.slice/crio-ee581d67aa7fb40f5d49f4db6548d810cd6595a80272e8c2817f47a64508e21a WatchSource:0}: Error finding container ee581d67aa7fb40f5d49f4db6548d810cd6595a80272e8c2817f47a64508e21a: Status 404 returned error can't find the container with id ee581d67aa7fb40f5d49f4db6548d810cd6595a80272e8c2817f47a64508e21a Oct 30 13:04:34.614153 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.614122 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:04:34.614224 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.614189 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:04:34.614224 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:34.614220 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:04:35.577433 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:35.577398 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" event={"ID":"c1423ef5-6c20-40a5-a870-4934b313ba4c","Type":"ContainerStarted","Data":"eb30591e2d53cb3587c35832487f18850ea2664b090f887e370a3fa29d3527db"} Oct 30 13:04:35.577862 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:35.577449 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" event={"ID":"c1423ef5-6c20-40a5-a870-4934b313ba4c","Type":"ContainerStarted","Data":"ee581d67aa7fb40f5d49f4db6548d810cd6595a80272e8c2817f47a64508e21a"} Oct 30 13:04:35.637335 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:35.637266 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" podStartSLOduration=2.637248406 podStartE2EDuration="2.637248406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:35.636975481 +0000 UTC m=+769.460203758" watchObservedRunningTime="2026-10-30 13:04:35.637248406 +0000 UTC m=+769.460476685" Oct 30 13:04:36.276742 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:36.275206 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 13:04:36.276742 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:36.275282 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" podUID="c1423ef5-6c20-40a5-a870-4934b313ba4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 13:04:36.276742 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:36.275636 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:37.273383 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:37.273344 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 13:04:37.273914 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:37.273411 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" podUID="c1423ef5-6c20-40a5-a870-4934b313ba4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 13:04:38.273385 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:38.273346 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 13:04:38.273598 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:38.273418 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" podUID="c1423ef5-6c20-40a5-a870-4934b313ba4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 13:04:39.273403 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:39.273362 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 13:04:39.273690 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:39.273662 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" podUID="c1423ef5-6c20-40a5-a870-4934b313ba4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 13:04:40.273361 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:40.273320 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 13:04:40.273638 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:40.273606 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" podUID="c1423ef5-6c20-40a5-a870-4934b313ba4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 13:04:41.484740 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:41.484703 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:41.486000 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:41.485975 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:41.487636 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:41.487609 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:04:49.387710 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.387663 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj"] Oct 30 13:04:49.391028 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.391004 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.411230 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.411209 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-175831-openshift-default-dockercfg-whpw8\"" Oct 30 13:04:49.425534 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.425496 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj"] Oct 30 13:04:49.461427 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.461057 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.461427 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.461107 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.461427 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.461136 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.461427 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.461185 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.461427 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.461226 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.461801 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.461769 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.461927 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.461863 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.462017 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.461955 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks98m\" (UniqueName: \"kubernetes.io/projected/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-kube-api-access-ks98m\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.462017 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.462002 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.563464 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.563420 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.563660 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.563478 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.563660 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.563507 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.563660 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.563542 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.563660 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.563582 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.563660 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.563628 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.563921 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.563666 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.563921 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.563726 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks98m\" (UniqueName: \"kubernetes.io/projected/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-kube-api-access-ks98m\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.563921 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.563769 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.564124 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.563967 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.564207 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.564180 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.564281 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.564254 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.564597 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.564574 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.564719 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.564617 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.566171 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.566148 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.566600 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.566580 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.591985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.591922 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.591985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.591977 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks98m\" (UniqueName: \"kubernetes.io/projected/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-kube-api-access-ks98m\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.703121 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.703078 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:49.842967 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.842919 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw"] Oct 30 13:04:49.846722 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.846698 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:49.863715 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.863692 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-bb71e2-openshift-default-dockercfg-qmj6k\"" Oct 30 13:04:49.880099 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.880068 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw"] Oct 30 13:04:49.970327 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.970239 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:49.970327 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.970287 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:49.970531 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.970398 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:49.970584 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.970533 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k569z\" (UniqueName: \"kubernetes.io/projected/b8e7baf7-ef26-41d4-9dec-f4987791862d-kube-api-access-k569z\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:49.970643 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.970618 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:49.970703 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.970645 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:49.970703 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.970676 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:49.970801 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.970755 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:49.970849 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:49.970833 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8e7baf7-ef26-41d4-9dec-f4987791862d-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.033742 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.033714 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj"] Oct 30 13:04:50.035524 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:04:50.035497 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c024b5b_a00f_4c31_bd17_b9fb2ee58b0b.slice/crio-ef895a030e1fea22a7e1be93e8939f7e0be07de2d3f1b36f0235ebe62f077076 WatchSource:0}: Error finding container ef895a030e1fea22a7e1be93e8939f7e0be07de2d3f1b36f0235ebe62f077076: Status 404 returned error can't find the container with id ef895a030e1fea22a7e1be93e8939f7e0be07de2d3f1b36f0235ebe62f077076 Oct 30 13:04:50.038481 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.038443 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:04:50.038551 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.038530 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:04:50.038602 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.038576 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:04:50.072302 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.072267 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.072433 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.072314 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.072433 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.072340 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.072433 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.072408 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.072793 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.072477 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8e7baf7-ef26-41d4-9dec-f4987791862d-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.072793 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.072539 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.072793 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.072569 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.072793 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.072611 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.072793 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.072689 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k569z\" (UniqueName: \"kubernetes.io/projected/b8e7baf7-ef26-41d4-9dec-f4987791862d-kube-api-access-k569z\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.072793 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.072733 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.073409 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.073375 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8e7baf7-ef26-41d4-9dec-f4987791862d-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.073987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.073916 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.074086 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.074057 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.074391 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.074357 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.075285 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.075265 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.076411 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.076388 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.101376 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.101354 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k569z\" (UniqueName: \"kubernetes.io/projected/b8e7baf7-ef26-41d4-9dec-f4987791862d-kube-api-access-k569z\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.102119 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.102094 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.161961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.160930 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:50.543392 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.543355 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw"] Oct 30 13:04:50.544504 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:04:50.544470 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e7baf7_ef26_41d4_9dec_f4987791862d.slice/crio-0166cedfa35e32f22096a833162dff0c01cc1ec211c6dc25de7f2d8e8eed26f1 WatchSource:0}: Error finding container 0166cedfa35e32f22096a833162dff0c01cc1ec211c6dc25de7f2d8e8eed26f1: Status 404 returned error can't find the container with id 0166cedfa35e32f22096a833162dff0c01cc1ec211c6dc25de7f2d8e8eed26f1 Oct 30 13:04:50.547099 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.547066 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:04:50.547205 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.547134 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:04:50.547205 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.547160 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:04:50.690009 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.689971 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" event={"ID":"b8e7baf7-ef26-41d4-9dec-f4987791862d","Type":"ContainerStarted","Data":"0ba68ef1b81e853d282700ae458baef67793bc3b8b558c1f749a9a56885662f0"} Oct 30 13:04:50.690179 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.690035 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" event={"ID":"b8e7baf7-ef26-41d4-9dec-f4987791862d","Type":"ContainerStarted","Data":"0166cedfa35e32f22096a833162dff0c01cc1ec211c6dc25de7f2d8e8eed26f1"} Oct 30 13:04:50.692310 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.692276 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" event={"ID":"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b","Type":"ContainerStarted","Data":"f9ae286681bf0522e9e08b38fe907de5a342de5a8a0e59bc8469c9a5a8393f9c"} Oct 30 13:04:50.692466 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.692330 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" event={"ID":"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b","Type":"ContainerStarted","Data":"ef895a030e1fea22a7e1be93e8939f7e0be07de2d3f1b36f0235ebe62f077076"} Oct 30 13:04:50.704243 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.704203 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:50.741791 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.741739 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" podStartSLOduration=1.7417198809999999 podStartE2EDuration="1.741719881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:50.740441928 +0000 UTC m=+784.563670205" watchObservedRunningTime="2026-10-30 13:04:50.741719881 +0000 UTC m=+784.564948229" Oct 30 13:04:50.788006 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:50.787919 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" podStartSLOduration=1.78790097 podStartE2EDuration="1.78790097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:50.787280473 +0000 UTC m=+784.610508763" watchObservedRunningTime="2026-10-30 13:04:50.78790097 +0000 UTC m=+784.611129248" Oct 30 13:04:51.173960 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:51.163102 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:51.173960 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:51.163279 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:04:51.173960 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:51.163325 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" podUID="b8e7baf7-ef26-41d4-9dec-f4987791862d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 13:04:51.713316 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:51.713274 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 13:04:51.713683 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:51.713349 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" podUID="3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 13:04:52.162361 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:52.162322 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:04:52.162656 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:52.162626 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" podUID="b8e7baf7-ef26-41d4-9dec-f4987791862d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 13:04:52.704543 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:52.704501 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 13:04:52.704731 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:52.704558 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" podUID="3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 13:04:53.161723 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:53.161687 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:04:53.162365 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:53.162338 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" podUID="b8e7baf7-ef26-41d4-9dec-f4987791862d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 13:04:53.710975 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:53.707931 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 13:04:53.710975 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:53.708043 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" podUID="3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 13:04:54.163963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:54.163381 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:04:54.163963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:54.163440 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" podUID="b8e7baf7-ef26-41d4-9dec-f4987791862d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 13:04:54.720108 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:54.720066 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 13:04:54.720437 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:54.720398 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" podUID="3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 13:04:55.196984 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:55.177531 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:04:55.196984 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:55.177588 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" podUID="b8e7baf7-ef26-41d4-9dec-f4987791862d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 13:04:55.705966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:55.704422 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 13:04:55.705966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:55.704494 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" podUID="3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 13:04:56.164961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:56.164351 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:04:56.164961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:56.164426 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" podUID="b8e7baf7-ef26-41d4-9dec-f4987791862d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 13:04:56.711243 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:56.710740 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 13:04:56.711243 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:56.710799 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" podUID="3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 13:04:57.162822 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:57.162783 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:04:57.163170 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:57.163137 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" podUID="b8e7baf7-ef26-41d4-9dec-f4987791862d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 13:04:58.683798 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:58.683758 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:58.685373 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:58.685349 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:58.686391 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:58.686369 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:04:59.034292 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:59.034211 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:59.034508 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:59.034476 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:59.044660 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:04:59.044499 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:04:59.703029 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:04:59.702977 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:05:02.564893 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:02.564852 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7"] Oct 30 13:05:02.568903 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:02.568877 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7" Oct 30 13:05:02.590464 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:02.590427 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7"] Oct 30 13:05:02.598013 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:02.597919 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:05:02.598212 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:02.598036 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:05:02.675166 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:02.675120 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6qv7\" (UniqueName: \"kubernetes.io/projected/71e7cebb-eea5-44c2-a772-89c46a9b63e2-kube-api-access-z6qv7\") pod \"maas-api-revoke-sub-7cb89605-c5gt7\" (UID: \"71e7cebb-eea5-44c2-a772-89c46a9b63e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7" Oct 30 13:05:02.675381 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:02.675199 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e7cebb-eea5-44c2-a772-89c46a9b63e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cb89605-c5gt7\" (UID: \"71e7cebb-eea5-44c2-a772-89c46a9b63e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7" Oct 30 13:05:02.675381 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:02.675287 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e7cebb-eea5-44c2-a772-89c46a9b63e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cb89605-c5gt7\" (UID: \"71e7cebb-eea5-44c2-a772-89c46a9b63e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7" Oct 30 13:05:02.776276 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:02.776241 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6qv7\" (UniqueName: \"kubernetes.io/projected/71e7cebb-eea5-44c2-a772-89c46a9b63e2-kube-api-access-z6qv7\") pod \"maas-api-revoke-sub-7cb89605-c5gt7\" (UID: \"71e7cebb-eea5-44c2-a772-89c46a9b63e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7" Oct 30 13:05:02.776446 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:02.776389 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e7cebb-eea5-44c2-a772-89c46a9b63e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cb89605-c5gt7\" (UID: \"71e7cebb-eea5-44c2-a772-89c46a9b63e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7" Oct 30 13:05:02.776446 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:02.776411 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e7cebb-eea5-44c2-a772-89c46a9b63e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cb89605-c5gt7\" (UID: \"71e7cebb-eea5-44c2-a772-89c46a9b63e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7" Oct 30 13:05:02.777041 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:02.777019 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e7cebb-eea5-44c2-a772-89c46a9b63e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cb89605-c5gt7\" (UID: \"71e7cebb-eea5-44c2-a772-89c46a9b63e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7" Oct 30 13:05:02.778792 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:02.778769 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e7cebb-eea5-44c2-a772-89c46a9b63e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cb89605-c5gt7\" (UID: \"71e7cebb-eea5-44c2-a772-89c46a9b63e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7" Oct 30 13:05:02.801654 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:02.801616 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6qv7\" (UniqueName: \"kubernetes.io/projected/71e7cebb-eea5-44c2-a772-89c46a9b63e2-kube-api-access-z6qv7\") pod \"maas-api-revoke-sub-7cb89605-c5gt7\" (UID: \"71e7cebb-eea5-44c2-a772-89c46a9b63e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7" Oct 30 13:05:02.883372 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:02.883290 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7" Oct 30 13:05:03.192514 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:03.192482 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7"] Oct 30 13:05:03.193903 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:05:03.193871 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e7cebb_eea5_44c2_a772_89c46a9b63e2.slice/crio-278820a84dc44c8f3aa61a2a2d9f252aee13cf6bad4e4dd85e306779ee31678a WatchSource:0}: Error finding container 278820a84dc44c8f3aa61a2a2d9f252aee13cf6bad4e4dd85e306779ee31678a: Status 404 returned error can't find the container with id 278820a84dc44c8f3aa61a2a2d9f252aee13cf6bad4e4dd85e306779ee31678a Oct 30 13:05:03.939560 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:03.939515 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7" event={"ID":"71e7cebb-eea5-44c2-a772-89c46a9b63e2","Type":"ContainerStarted","Data":"278820a84dc44c8f3aa61a2a2d9f252aee13cf6bad4e4dd85e306779ee31678a"} Oct 30 13:05:05.957343 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:05.957297 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7" event={"ID":"71e7cebb-eea5-44c2-a772-89c46a9b63e2","Type":"ContainerStarted","Data":"a627a0be781883afd17a75bd0f9add6d3f6207262e6f300161418c626d7afc23"} Oct 30 13:05:06.002239 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:06.002177 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7" podStartSLOduration=1.7269122430000001 podStartE2EDuration="4.002160899s" totalImagesPullingTime="2.275248656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:05.99878632 +0000 UTC m=+799.822014607" watchObservedRunningTime="2026-10-30 13:05:06.002160899 +0000 UTC m=+799.825389175" Oct 30 13:05:06.966825 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:06.966786 4503 generic.go:356] "Generic (PLEG): container finished" podID="71e7cebb-eea5-44c2-a772-89c46a9b63e2" containerID="a627a0be781883afd17a75bd0f9add6d3f6207262e6f300161418c626d7afc23" exitCode=0 Oct 30 13:05:06.967350 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:06.966998 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7" event={"ID":"71e7cebb-eea5-44c2-a772-89c46a9b63e2","Type":"ContainerDied","Data":"a627a0be781883afd17a75bd0f9add6d3f6207262e6f300161418c626d7afc23"} Oct 30 13:05:08.430475 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:08.430438 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7" Oct 30 13:05:08.470417 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:08.467731 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71e7cebb-eea5-44c2-a772-89c46a9b63e2-kube-api-access-z6qv7\" (UniqueName: \"kubernetes.io/projected/71e7cebb-eea5-44c2-a772-89c46a9b63e2-kube-api-access-z6qv7\") pod \"71e7cebb-eea5-44c2-a772-89c46a9b63e2\" (UID: \"71e7cebb-eea5-44c2-a772-89c46a9b63e2\") " Oct 30 13:05:08.470417 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:08.468258 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71e7cebb-eea5-44c2-a772-89c46a9b63e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e7cebb-eea5-44c2-a772-89c46a9b63e2-maas-api-xks-ca\") pod \"71e7cebb-eea5-44c2-a772-89c46a9b63e2\" (UID: \"71e7cebb-eea5-44c2-a772-89c46a9b63e2\") " Oct 30 13:05:08.470417 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:08.468719 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71e7cebb-eea5-44c2-a772-89c46a9b63e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e7cebb-eea5-44c2-a772-89c46a9b63e2-maas-api-openshift-ca\") pod \"71e7cebb-eea5-44c2-a772-89c46a9b63e2\" (UID: \"71e7cebb-eea5-44c2-a772-89c46a9b63e2\") " Oct 30 13:05:08.470417 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:08.469518 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e7cebb-eea5-44c2-a772-89c46a9b63e2-maas-api-openshift-ca" pod "71e7cebb-eea5-44c2-a772-89c46a9b63e2" (UID: "71e7cebb-eea5-44c2-a772-89c46a9b63e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:08.470908 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:08.470870 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e7cebb-eea5-44c2-a772-89c46a9b63e2-kube-api-access-z6qv7" pod "71e7cebb-eea5-44c2-a772-89c46a9b63e2" (UID: "71e7cebb-eea5-44c2-a772-89c46a9b63e2"). InnerVolumeSpecName "kube-api-access-z6qv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:08.472180 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:08.472155 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e7cebb-eea5-44c2-a772-89c46a9b63e2-maas-api-xks-ca" pod "71e7cebb-eea5-44c2-a772-89c46a9b63e2" (UID: "71e7cebb-eea5-44c2-a772-89c46a9b63e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:08.591982 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:08.591928 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e7cebb-eea5-44c2-a772-89c46a9b63e2-maas-api-openshift-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:05:08.592218 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:08.592198 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6qv7\" (UniqueName: \"kubernetes.io/projected/71e7cebb-eea5-44c2-a772-89c46a9b63e2-kube-api-access-z6qv7\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:05:08.592331 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:08.592318 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e7cebb-eea5-44c2-a772-89c46a9b63e2-maas-api-xks-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:05:09.015572 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:09.015523 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7" event={"ID":"71e7cebb-eea5-44c2-a772-89c46a9b63e2","Type":"ContainerDied","Data":"278820a84dc44c8f3aa61a2a2d9f252aee13cf6bad4e4dd85e306779ee31678a"} Oct 30 13:05:09.015886 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:09.015865 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="278820a84dc44c8f3aa61a2a2d9f252aee13cf6bad4e4dd85e306779ee31678a" Oct 30 13:05:09.016117 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:09.016100 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7" Oct 30 13:05:19.256489 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.256447 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm"] Oct 30 13:05:19.257391 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.257367 4503 cpu_manager.go:414] "containerMap: removing container" podUID="71e7cebb-eea5-44c2-a772-89c46a9b63e2" containerName="revoke-keys" Oct 30 13:05:19.257441 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.257395 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e7cebb-eea5-44c2-a772-89c46a9b63e2" containerName="revoke-keys" Oct 30 13:05:19.257587 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.257573 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="71e7cebb-eea5-44c2-a772-89c46a9b63e2" containerName="revoke-keys" Oct 30 13:05:19.261870 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.261847 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.286344 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.286306 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm"] Oct 30 13:05:19.296457 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.295913 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-175831\"/\"default-dockercfg-png5h\"" Oct 30 13:05:19.296457 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.296311 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-175831\"/\"openshift-service-ca.crt\"" Oct 30 13:05:19.296706 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.296392 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-175831\"/\"kube-root-ca.crt\"" Oct 30 13:05:19.297121 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.297101 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-175831\"/\"distinct-w5-175831-kserve-self-signed-certs\"" Oct 30 13:05:19.327859 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.327813 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-model-cache\") pod \"distinct-w5-175831-kserve-d98f98cfc-vvzpm\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.328088 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.327933 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p54x4\" (UniqueName: \"kubernetes.io/projected/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-kube-api-access-p54x4\") pod \"distinct-w5-175831-kserve-d98f98cfc-vvzpm\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.328163 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.328083 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-dshm\") pod \"distinct-w5-175831-kserve-d98f98cfc-vvzpm\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.328232 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.328178 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-tls-certs\") pod \"distinct-w5-175831-kserve-d98f98cfc-vvzpm\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.328315 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.328294 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-tmp-dir\") pod \"distinct-w5-175831-kserve-d98f98cfc-vvzpm\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.328378 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.328333 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-home\") pod \"distinct-w5-175831-kserve-d98f98cfc-vvzpm\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.429618 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.429580 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-tls-certs\") pod \"distinct-w5-175831-kserve-d98f98cfc-vvzpm\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.429819 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.429632 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-tmp-dir\") pod \"distinct-w5-175831-kserve-d98f98cfc-vvzpm\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.429819 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.429657 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-home\") pod \"distinct-w5-175831-kserve-d98f98cfc-vvzpm\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.429819 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.429689 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-model-cache\") pod \"distinct-w5-175831-kserve-d98f98cfc-vvzpm\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.429819 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.429729 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p54x4\" (UniqueName: \"kubernetes.io/projected/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-kube-api-access-p54x4\") pod \"distinct-w5-175831-kserve-d98f98cfc-vvzpm\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.429819 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.429792 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-dshm\") pod \"distinct-w5-175831-kserve-d98f98cfc-vvzpm\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.430318 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.430240 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-home\") pod \"distinct-w5-175831-kserve-d98f98cfc-vvzpm\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.430318 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.430240 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-tmp-dir\") pod \"distinct-w5-175831-kserve-d98f98cfc-vvzpm\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.430470 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.430315 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-model-cache\") pod \"distinct-w5-175831-kserve-d98f98cfc-vvzpm\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.432050 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.432023 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-dshm\") pod \"distinct-w5-175831-kserve-d98f98cfc-vvzpm\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.432220 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.432203 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-tls-certs\") pod \"distinct-w5-175831-kserve-d98f98cfc-vvzpm\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.455757 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.455728 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p54x4\" (UniqueName: \"kubernetes.io/projected/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-kube-api-access-p54x4\") pod \"distinct-w5-175831-kserve-d98f98cfc-vvzpm\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.574478 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.574376 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:19.951963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:19.921310 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m"] Oct 30 13:05:20.335778 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.291306 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m"] Oct 30 13:05:20.384181 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.384141 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm"] Oct 30 13:05:20.389768 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.301097 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.497990 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.487712 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788f38f0-c8b1-461b-969e-8d171ad33149-istiod-ca-cert\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.507211 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.507097 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788f38f0-c8b1-461b-969e-8d171ad33149-istio-podinfo\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.507642 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.507543 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-workload-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.507843 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.507813 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-workload-certs\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.521190 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.521157 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtbzg\" (UniqueName: \"kubernetes.io/projected/788f38f0-c8b1-461b-969e-8d171ad33149-kube-api-access-wtbzg\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.521596 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.521572 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788f38f0-c8b1-461b-969e-8d171ad33149-istio-token\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.521951 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.521916 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-istio-data\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.523406 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.522181 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-credential-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.546625 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.546580 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-istio-envoy\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.650371 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.650322 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-istio-data\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.660491 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.660448 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-credential-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.660794 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.660754 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-istio-envoy\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.661423 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.661382 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788f38f0-c8b1-461b-969e-8d171ad33149-istiod-ca-cert\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.661626 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.661598 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788f38f0-c8b1-461b-969e-8d171ad33149-istio-podinfo\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.682542 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.681928 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788f38f0-c8b1-461b-969e-8d171ad33149-istiod-ca-cert\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.682542 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.655108 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-istio-data\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.682542 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.682479 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-credential-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.684069 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.684047 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-workload-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.684292 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.684264 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-workload-certs\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.684522 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.684504 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtbzg\" (UniqueName: \"kubernetes.io/projected/788f38f0-c8b1-461b-969e-8d171ad33149-kube-api-access-wtbzg\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.684747 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.684726 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788f38f0-c8b1-461b-969e-8d171ad33149-istio-token\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.686460 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.686436 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-workload-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.686932 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.686911 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-workload-certs\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.739652 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.739575 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-istio-envoy\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.803321 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.778450 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788f38f0-c8b1-461b-969e-8d171ad33149-istio-token\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.803321 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.778663 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtbzg\" (UniqueName: \"kubernetes.io/projected/788f38f0-c8b1-461b-969e-8d171ad33149-kube-api-access-wtbzg\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.803321 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.785887 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788f38f0-c8b1-461b-969e-8d171ad33149-istio-podinfo\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:20.863471 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:20.863192 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:21.507141 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:21.482456 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" event={"ID":"0e3ae60c-3a8a-439f-877e-ea7e8e890b11","Type":"ContainerStarted","Data":"3769555a6057686e0154b6852bb172107515c671fdf7f856c5fa7e87654321de"} Oct 30 13:05:21.507141 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:21.482513 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" event={"ID":"0e3ae60c-3a8a-439f-877e-ea7e8e890b11","Type":"ContainerStarted","Data":"88e7e98a6d52242ba6c8a312494633534c317fb5d25ea8db6f9b9b929b01ce5a"} Oct 30 13:05:21.576065 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:21.576030 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:21.618874 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:21.618793 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podStartSLOduration=2.618773807 podStartE2EDuration="2.618773807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:21.616240363 +0000 UTC m=+815.439468641" watchObservedRunningTime="2026-10-30 13:05:21.618773807 +0000 UTC m=+815.442002091" Oct 30 13:05:21.999657 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:21.998322 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m"] Oct 30 13:05:22.184218 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:22.184146 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:22.184668 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:22.184634 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:22.213958 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:22.209469 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:22.623318 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:22.623281 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" event={"ID":"788f38f0-c8b1-461b-969e-8d171ad33149","Type":"ContainerStarted","Data":"974c12b0d757d6cfcd3bf8212265dca79509979f64cf3f513c946270255dbe1a"} Oct 30 13:05:23.640223 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:23.640186 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" event={"ID":"788f38f0-c8b1-461b-969e-8d171ad33149","Type":"ContainerStarted","Data":"b4663661b422579d5b7c09addfc9808b6aad52711328bc82dc422abd2b326947"} Oct 30 13:05:23.727959 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:23.724726 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" podStartSLOduration=4.724705973 podStartE2EDuration="4.724705973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:23.724218019 +0000 UTC m=+817.547446300" watchObservedRunningTime="2026-10-30 13:05:23.724705973 +0000 UTC m=+817.547934251" Oct 30 13:05:23.866691 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:23.866655 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:23.867092 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:23.867066 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 13:05:23.867283 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:23.867257 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" podUID="788f38f0-c8b1-461b-969e-8d171ad33149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 13:05:24.874137 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:24.874088 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 13:05:24.874676 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:24.874181 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" podUID="788f38f0-c8b1-461b-969e-8d171ad33149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 13:05:25.865427 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:25.865386 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 13:05:25.865697 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:25.865661 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" podUID="788f38f0-c8b1-461b-969e-8d171ad33149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 13:05:26.865965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:26.865365 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 13:05:26.865965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:26.865430 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" podUID="788f38f0-c8b1-461b-969e-8d171ad33149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 13:05:27.865962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:27.865560 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 13:05:27.865962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:27.865627 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" podUID="788f38f0-c8b1-461b-969e-8d171ad33149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 13:05:28.866346 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:28.866307 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 13:05:28.866951 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:28.866899 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" podUID="788f38f0-c8b1-461b-969e-8d171ad33149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 13:05:29.575204 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:29.575164 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:29.588797 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:29.588761 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:29.702518 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:05:29.702475 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:05:29.719933 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:29.719903 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:05:29.865632 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:29.865539 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 13:05:29.865632 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:29.865598 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" podUID="788f38f0-c8b1-461b-969e-8d171ad33149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 13:05:30.870242 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:30.870209 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:30.870651 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:30.870479 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:30.871342 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:30.871322 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:05:35.074961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.071502 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29"] Oct 30 13:05:35.240002 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.239917 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29"] Oct 30 13:05:35.240222 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.240081 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.267454 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.267418 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv"] Oct 30 13:05:35.272303 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.272280 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.300461 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.300426 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv"] Oct 30 13:05:35.338541 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.338439 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.338541 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.338504 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.338541 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.338530 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88155745-668a-4d1c-8114-2c9eb2a13dba-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.338973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.338573 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.338973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.338605 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.338973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.338663 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.338973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.338690 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51742836-262a-4345-ad56-bb8725b209ba-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.338973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.338715 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.338973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.338760 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.338973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.338788 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.338973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.338847 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.338973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.338875 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jfvz\" (UniqueName: \"kubernetes.io/projected/51742836-262a-4345-ad56-bb8725b209ba-kube-api-access-6jfvz\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.338973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.338906 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51742836-262a-4345-ad56-bb8725b209ba-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.339684 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.339001 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.339684 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.339044 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlrnh\" (UniqueName: \"kubernetes.io/projected/88155745-668a-4d1c-8114-2c9eb2a13dba-kube-api-access-nlrnh\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.339684 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.339112 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.339684 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.339143 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.339684 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.339177 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51742836-262a-4345-ad56-bb8725b209ba-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.440630 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.440588 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.440822 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.440646 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.440822 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.440690 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51742836-262a-4345-ad56-bb8725b209ba-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.440822 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.440780 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.441104 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.440831 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.441104 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.440864 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88155745-668a-4d1c-8114-2c9eb2a13dba-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.441104 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.440913 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.441104 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.440987 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.441104 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.441035 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.441104 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.441064 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51742836-262a-4345-ad56-bb8725b209ba-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.441104 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.441092 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.441445 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.441139 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.441445 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.441145 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.441445 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.441219 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.441445 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.441230 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.441445 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.441318 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.441445 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.441358 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jfvz\" (UniqueName: \"kubernetes.io/projected/51742836-262a-4345-ad56-bb8725b209ba-kube-api-access-6jfvz\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.441445 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.441395 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51742836-262a-4345-ad56-bb8725b209ba-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.441445 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.441440 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.442070 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.441734 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.442070 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.441446 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.442070 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.441886 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlrnh\" (UniqueName: \"kubernetes.io/projected/88155745-668a-4d1c-8114-2c9eb2a13dba-kube-api-access-nlrnh\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.442298 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.442127 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51742836-262a-4345-ad56-bb8725b209ba-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.442298 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.442220 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88155745-668a-4d1c-8114-2c9eb2a13dba-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.442404 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.442370 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.442647 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.442625 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.442647 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.441819 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.442933 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.442909 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.444135 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.444110 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.444993 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.444966 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.445076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.444997 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51742836-262a-4345-ad56-bb8725b209ba-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.445076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.444970 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.466608 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.466564 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51742836-262a-4345-ad56-bb8725b209ba-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.466767 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.466712 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jfvz\" (UniqueName: \"kubernetes.io/projected/51742836-262a-4345-ad56-bb8725b209ba-kube-api-access-6jfvz\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.467383 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.467357 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlrnh\" (UniqueName: \"kubernetes.io/projected/88155745-668a-4d1c-8114-2c9eb2a13dba-kube-api-access-nlrnh\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.467599 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.467563 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.566661 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.566604 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:35.596073 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.595994 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:35.927714 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:05:35.927675 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88155745_668a_4d1c_8114_2c9eb2a13dba.slice/crio-15d4d353f490ae8b3899d91261d63899c698842eb20c36256a496afa1c72224c WatchSource:0}: Error finding container 15d4d353f490ae8b3899d91261d63899c698842eb20c36256a496afa1c72224c: Status 404 returned error can't find the container with id 15d4d353f490ae8b3899d91261d63899c698842eb20c36256a496afa1c72224c Oct 30 13:05:35.928599 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.928558 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29"] Oct 30 13:05:35.931132 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.931088 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:35.931239 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.931173 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:35.931239 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:35.931217 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:36.027976 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:36.027924 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv"] Oct 30 13:05:36.037981 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:05:36.037904 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51742836_262a_4345_ad56_bb8725b209ba.slice/crio-91ee2e8e4584535a776bd5cc6a418c238be7c48174e69310cff7be3be57c576d WatchSource:0}: Error finding container 91ee2e8e4584535a776bd5cc6a418c238be7c48174e69310cff7be3be57c576d: Status 404 returned error can't find the container with id 91ee2e8e4584535a776bd5cc6a418c238be7c48174e69310cff7be3be57c576d Oct 30 13:05:36.041520 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:36.041479 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:36.041616 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:36.041571 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:36.041655 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:36.041615 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:36.811702 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:36.811658 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" event={"ID":"88155745-668a-4d1c-8114-2c9eb2a13dba","Type":"ContainerStarted","Data":"a4c7b83daa8ef389d5976f326928a58d0fcbb4c86cffaf6c5bf486d82e71a448"} Oct 30 13:05:36.812422 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:36.812386 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" event={"ID":"88155745-668a-4d1c-8114-2c9eb2a13dba","Type":"ContainerStarted","Data":"15d4d353f490ae8b3899d91261d63899c698842eb20c36256a496afa1c72224c"} Oct 30 13:05:36.820087 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:36.820067 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" event={"ID":"51742836-262a-4345-ad56-bb8725b209ba","Type":"ContainerStarted","Data":"5352887cfd1c57622cf52ce5045f7a766c1a33d1662a56dd3533717120514ae3"} Oct 30 13:05:36.820302 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:36.820288 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" event={"ID":"51742836-262a-4345-ad56-bb8725b209ba","Type":"ContainerStarted","Data":"91ee2e8e4584535a776bd5cc6a418c238be7c48174e69310cff7be3be57c576d"} Oct 30 13:05:36.873963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:36.872456 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" podStartSLOduration=1.8724350969999999 podStartE2EDuration="1.872435097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:36.870442727 +0000 UTC m=+830.693671005" watchObservedRunningTime="2026-10-30 13:05:36.872435097 +0000 UTC m=+830.695663375" Oct 30 13:05:36.921124 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:36.921075 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" podStartSLOduration=1.921059472 podStartE2EDuration="1.921059472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:36.920212677 +0000 UTC m=+830.743440962" watchObservedRunningTime="2026-10-30 13:05:36.921059472 +0000 UTC m=+830.744287750" Oct 30 13:05:37.572965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:37.568357 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:37.593129 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:37.593083 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 13:05:37.593468 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:37.593437 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" podUID="88155745-668a-4d1c-8114-2c9eb2a13dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 13:05:37.597028 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:37.597007 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:37.598983 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:37.598905 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 13:05:37.598983 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:37.598963 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" podUID="51742836-262a-4345-ad56-bb8725b209ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 13:05:38.569965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:38.567374 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 13:05:38.569965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:38.567458 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" podUID="88155745-668a-4d1c-8114-2c9eb2a13dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 13:05:38.596649 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:38.596611 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 13:05:38.596995 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:38.596931 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" podUID="51742836-262a-4345-ad56-bb8725b209ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 13:05:39.168580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.168090 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s"] Oct 30 13:05:39.194378 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.188544 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.227064 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.226414 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-bb71e2\"/\"kube-root-ca.crt\"" Oct 30 13:05:39.231777 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.229351 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s"] Oct 30 13:05:39.236811 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.235783 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-bb71e2\"/\"openshift-service-ca.crt\"" Oct 30 13:05:39.236811 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.235896 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb71e2\"/\"distinct-w2-bb71e2-kserve-self-signed-certs\"" Oct 30 13:05:39.236811 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.236032 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb71e2\"/\"default-dockercfg-8lldv\"" Oct 30 13:05:39.316478 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.316440 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-home\") pod \"distinct-w2-bb71e2-kserve-7f4b84967b-24r9s\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.317263 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.316704 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-dshm\") pod \"distinct-w2-bb71e2-kserve-7f4b84967b-24r9s\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.317263 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.316886 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfdbr\" (UniqueName: \"kubernetes.io/projected/2001219c-7af6-4dab-a7cc-ec2a30186f2e-kube-api-access-dfdbr\") pod \"distinct-w2-bb71e2-kserve-7f4b84967b-24r9s\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.317263 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.316962 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2001219c-7af6-4dab-a7cc-ec2a30186f2e-tls-certs\") pod \"distinct-w2-bb71e2-kserve-7f4b84967b-24r9s\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.317263 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.317065 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-model-cache\") pod \"distinct-w2-bb71e2-kserve-7f4b84967b-24r9s\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.317263 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.317087 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-tmp-dir\") pod \"distinct-w2-bb71e2-kserve-7f4b84967b-24r9s\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.467011 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.463731 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfdbr\" (UniqueName: \"kubernetes.io/projected/2001219c-7af6-4dab-a7cc-ec2a30186f2e-kube-api-access-dfdbr\") pod \"distinct-w2-bb71e2-kserve-7f4b84967b-24r9s\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.467011 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.463802 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2001219c-7af6-4dab-a7cc-ec2a30186f2e-tls-certs\") pod \"distinct-w2-bb71e2-kserve-7f4b84967b-24r9s\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.467011 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.463891 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-model-cache\") pod \"distinct-w2-bb71e2-kserve-7f4b84967b-24r9s\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.467011 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.463913 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-tmp-dir\") pod \"distinct-w2-bb71e2-kserve-7f4b84967b-24r9s\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.467011 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.464002 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-home\") pod \"distinct-w2-bb71e2-kserve-7f4b84967b-24r9s\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.467011 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.464025 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-dshm\") pod \"distinct-w2-bb71e2-kserve-7f4b84967b-24r9s\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.534442 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.532006 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6"] Oct 30 13:05:39.557978 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.552636 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-home\") pod \"distinct-w2-bb71e2-kserve-7f4b84967b-24r9s\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.557978 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.555153 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-tmp-dir\") pod \"distinct-w2-bb71e2-kserve-7f4b84967b-24r9s\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.598965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.595195 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 13:05:39.598965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.595257 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" podUID="88155745-668a-4d1c-8114-2c9eb2a13dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 13:05:39.623758 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.620645 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.623758 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.621128 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 13:05:39.623758 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.621230 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" podUID="51742836-262a-4345-ad56-bb8725b209ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 13:05:39.623758 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.623536 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6"] Oct 30 13:05:39.625187 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.625153 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-model-cache\") pod \"distinct-w2-bb71e2-kserve-7f4b84967b-24r9s\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.635187 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.635154 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-dshm\") pod \"distinct-w2-bb71e2-kserve-7f4b84967b-24r9s\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.728580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.724658 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2001219c-7af6-4dab-a7cc-ec2a30186f2e-tls-certs\") pod \"distinct-w2-bb71e2-kserve-7f4b84967b-24r9s\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.728580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.725883 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfdbr\" (UniqueName: \"kubernetes.io/projected/2001219c-7af6-4dab-a7cc-ec2a30186f2e-kube-api-access-dfdbr\") pod \"distinct-w2-bb71e2-kserve-7f4b84967b-24r9s\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.728580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.728238 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8ab9ac-098f-4b41-9f0d-51422627d481-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.728580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.728292 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.728580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.728348 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.728580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.728420 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.728580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.728455 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.729169 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.728608 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.729169 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.728662 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.729169 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.728717 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.729169 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.728790 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-285kq\" (UniqueName: \"kubernetes.io/projected/dc8ab9ac-098f-4b41-9f0d-51422627d481-kube-api-access-285kq\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.830635 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.830587 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8ab9ac-098f-4b41-9f0d-51422627d481-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.830635 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.830635 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.830910 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.830671 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.830910 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.830715 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.830910 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.830733 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.830910 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.830767 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.830910 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.830801 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.830910 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.830828 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.830910 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.830851 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-285kq\" (UniqueName: \"kubernetes.io/projected/dc8ab9ac-098f-4b41-9f0d-51422627d481-kube-api-access-285kq\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.831737 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.831712 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8ab9ac-098f-4b41-9f0d-51422627d481-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.832509 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.832483 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.832617 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.832518 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.832677 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.832623 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.833083 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.833058 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.834427 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.834404 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.834527 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.834459 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.899962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.899276 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:39.903172 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.902760 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-285kq\" (UniqueName: \"kubernetes.io/projected/dc8ab9ac-098f-4b41-9f0d-51422627d481-kube-api-access-285kq\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:39.903172 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:39.902954 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mbnm6\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:40.040698 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:40.040668 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:40.309641 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:40.309248 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s"] Oct 30 13:05:40.312318 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:05:40.312284 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2001219c_7af6_4dab_a7cc_ec2a30186f2e.slice/crio-9b3e9513630b5f2e7baff45f74d4975d8e6ae18d83f6c9a385771255e239dad6 WatchSource:0}: Error finding container 9b3e9513630b5f2e7baff45f74d4975d8e6ae18d83f6c9a385771255e239dad6: Status 404 returned error can't find the container with id 9b3e9513630b5f2e7baff45f74d4975d8e6ae18d83f6c9a385771255e239dad6 Oct 30 13:05:40.430988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:40.430960 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6"] Oct 30 13:05:40.434206 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:05:40.434172 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8ab9ac_098f_4b41_9f0d_51422627d481.slice/crio-e74a4b7f3bc8262c938dbdc93b1be746421cbd72ddf1d9959206c21eea7fa4b9 WatchSource:0}: Error finding container e74a4b7f3bc8262c938dbdc93b1be746421cbd72ddf1d9959206c21eea7fa4b9: Status 404 returned error can't find the container with id e74a4b7f3bc8262c938dbdc93b1be746421cbd72ddf1d9959206c21eea7fa4b9 Oct 30 13:05:40.436679 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:40.436649 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:40.436756 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:40.436712 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:40.436756 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:40.436739 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:40.583963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:40.573210 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 13:05:40.583963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:40.573289 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" podUID="88155745-668a-4d1c-8114-2c9eb2a13dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 13:05:40.598962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:40.596408 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 13:05:40.598962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:40.596470 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" podUID="51742836-262a-4345-ad56-bb8725b209ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 13:05:40.885021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:40.884358 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" event={"ID":"dc8ab9ac-098f-4b41-9f0d-51422627d481","Type":"ContainerStarted","Data":"0f240e34b87ea0291df9fef93f44dfc807ce3b33cbd42f0f63857773ce414673"} Oct 30 13:05:40.885021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:40.884432 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" event={"ID":"dc8ab9ac-098f-4b41-9f0d-51422627d481","Type":"ContainerStarted","Data":"e74a4b7f3bc8262c938dbdc93b1be746421cbd72ddf1d9959206c21eea7fa4b9"} Oct 30 13:05:40.888175 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:40.888143 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" event={"ID":"2001219c-7af6-4dab-a7cc-ec2a30186f2e","Type":"ContainerStarted","Data":"d22d0988438c833e247c32d3a009645393a14fbba89fbca9f85a3f5bcf4cdc19"} Oct 30 13:05:40.888391 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:40.888372 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" event={"ID":"2001219c-7af6-4dab-a7cc-ec2a30186f2e","Type":"ContainerStarted","Data":"9b3e9513630b5f2e7baff45f74d4975d8e6ae18d83f6c9a385771255e239dad6"} Oct 30 13:05:40.907050 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:40.907005 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:40.941963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:40.941341 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" podStartSLOduration=1.94132177 podStartE2EDuration="1.94132177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:40.938370715 +0000 UTC m=+834.761598993" watchObservedRunningTime="2026-10-30 13:05:40.94132177 +0000 UTC m=+834.764550048" Oct 30 13:05:41.007964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:41.003213 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podStartSLOduration=2.003189547 podStartE2EDuration="2.003189547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:41.000267817 +0000 UTC m=+834.823496135" watchObservedRunningTime="2026-10-30 13:05:41.003189547 +0000 UTC m=+834.826417825" Oct 30 13:05:41.040901 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:41.040869 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:41.043492 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:41.043466 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-mbnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 13:05:41.043660 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:41.043511 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" podUID="dc8ab9ac-098f-4b41-9f0d-51422627d481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 13:05:41.581964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:41.578371 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 13:05:41.581964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:41.578443 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" podUID="88155745-668a-4d1c-8114-2c9eb2a13dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 13:05:41.613964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:41.607346 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 13:05:41.613964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:41.607409 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" podUID="51742836-262a-4345-ad56-bb8725b209ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 13:05:42.041445 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:42.041402 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-mbnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 13:05:42.041660 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:42.041455 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" podUID="dc8ab9ac-098f-4b41-9f0d-51422627d481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 13:05:42.568963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:42.568315 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 13:05:42.568963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:42.568380 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" podUID="88155745-668a-4d1c-8114-2c9eb2a13dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 13:05:42.596957 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:42.596904 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 13:05:42.597165 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:42.597005 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" podUID="51742836-262a-4345-ad56-bb8725b209ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 13:05:43.044962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:43.044552 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-mbnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 13:05:43.044962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:43.044643 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" podUID="dc8ab9ac-098f-4b41-9f0d-51422627d481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 13:05:43.581302 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:43.581254 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:43.581672 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:43.581641 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:43.582787 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:43.582765 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:05:43.624311 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:43.624281 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:43.625428 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:43.625400 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:43.627346 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:43.627322 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:05:44.049961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:44.042159 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-mbnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 13:05:44.049961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:44.042224 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" podUID="dc8ab9ac-098f-4b41-9f0d-51422627d481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 13:05:45.041315 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:45.041283 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-mbnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 13:05:45.041490 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:45.041343 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" podUID="dc8ab9ac-098f-4b41-9f0d-51422627d481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 13:05:46.041483 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:46.041446 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-mbnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 13:05:46.041875 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:46.041520 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" podUID="dc8ab9ac-098f-4b41-9f0d-51422627d481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 13:05:47.042006 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:47.041964 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-mbnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 13:05:47.042397 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:47.042032 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" podUID="dc8ab9ac-098f-4b41-9f0d-51422627d481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 13:05:48.041916 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:48.041871 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-mbnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 13:05:48.042123 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:48.041934 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" podUID="dc8ab9ac-098f-4b41-9f0d-51422627d481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 13:05:49.335791 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.335744 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f"] Oct 30 13:05:49.364106 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.364054 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f"] Oct 30 13:05:49.384089 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.384056 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.469963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.444685 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.473294 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.473173 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.497248 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.497066 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbqlr\" (UniqueName: \"kubernetes.io/projected/c715c318-f465-4b5a-8c65-10c6539a5c61-kube-api-access-zbqlr\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.497674 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.497552 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.497990 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.497969 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.498328 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.498304 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c715c318-f465-4b5a-8c65-10c6539a5c61-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.498732 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.498710 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.498918 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.498889 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.499375 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.499257 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.601779 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.601691 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.602045 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.602018 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.602226 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.602209 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.602506 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.602488 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.602635 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.602617 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.602749 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.602734 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbqlr\" (UniqueName: \"kubernetes.io/projected/c715c318-f465-4b5a-8c65-10c6539a5c61-kube-api-access-zbqlr\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.602872 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.602856 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.603038 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.603021 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.603171 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.603153 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c715c318-f465-4b5a-8c65-10c6539a5c61-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.603823 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.603794 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.604085 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.604063 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.604406 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.604386 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c715c318-f465-4b5a-8c65-10c6539a5c61-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.605094 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.605072 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.605472 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.605450 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.623956 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.612367 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.624999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.624973 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.653965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.642276 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.653965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.642396 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbqlr\" (UniqueName: \"kubernetes.io/projected/c715c318-f465-4b5a-8c65-10c6539a5c61-kube-api-access-zbqlr\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-2r24f\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.849493 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.846265 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:49.901124 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.900215 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:49.921393 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:49.921091 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:50.083546 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:50.083084 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-mbnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:05:50.083546 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:50.083169 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" podUID="dc8ab9ac-098f-4b41-9f0d-51422627d481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:05:50.143134 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:50.143110 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:05:50.211561 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:50.211513 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f"] Oct 30 13:05:50.218009 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:50.217964 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:50.218132 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:50.218067 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:50.218132 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:50.218116 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:50.223984 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:50.220409 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:50.223984 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:50.222273 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:50.223984 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:50.223434 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:05:51.136915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:51.136840 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" event={"ID":"c715c318-f465-4b5a-8c65-10c6539a5c61","Type":"ContainerStarted","Data":"504c3f6fb85b53ecbe3eebb10ab3a6c972b6645431801a4e4c12671c503ea418"} Oct 30 13:05:51.137468 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:51.136923 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" event={"ID":"c715c318-f465-4b5a-8c65-10c6539a5c61","Type":"ContainerStarted","Data":"ee684c4cecc846e2d3028567117242ae6cb1cf0de37d9d23d0dcd862f1a161f7"} Oct 30 13:05:51.213302 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:51.213240 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" podStartSLOduration=2.21322173 podStartE2EDuration="2.21322173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:51.212383217 +0000 UTC m=+845.035611495" watchObservedRunningTime="2026-10-30 13:05:51.21322173 +0000 UTC m=+845.036450008" Oct 30 13:05:51.848360 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:51.848321 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:51.857530 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:51.857056 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 13:05:51.857530 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:51.857136 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" podUID="c715c318-f465-4b5a-8c65-10c6539a5c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 13:05:52.848382 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:52.848341 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 13:05:52.849019 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:52.848985 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" podUID="c715c318-f465-4b5a-8c65-10c6539a5c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 13:05:53.849369 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:53.849324 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 13:05:53.850000 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:53.849966 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" podUID="c715c318-f465-4b5a-8c65-10c6539a5c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 13:05:54.865394 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:54.862356 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 13:05:54.865394 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:54.862458 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" podUID="c715c318-f465-4b5a-8c65-10c6539a5c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 13:05:55.848400 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:55.848357 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 13:05:55.848720 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:55.848684 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" podUID="c715c318-f465-4b5a-8c65-10c6539a5c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 13:05:56.847039 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:56.846995 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 13:05:56.847555 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:56.847060 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" podUID="c715c318-f465-4b5a-8c65-10c6539a5c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 13:05:57.850968 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:57.850927 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:57.851449 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:57.851438 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:57.852121 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:05:57.852101 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:05:59.707412 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:05:59.707316 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:06:29.702194 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:06:29.702157 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:06:34.710113 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.710072 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh"] Oct 30 13:06:34.718269 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.718239 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.738533 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.738498 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh"] Oct 30 13:06:34.811948 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.811892 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.812119 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.812073 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.812166 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.812137 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.812216 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.812200 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.812252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.812230 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.812286 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.812270 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.812387 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.812368 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46pc6\" (UniqueName: \"kubernetes.io/projected/b0c97856-79d6-4471-bcf8-36b0bc4e5037-kube-api-access-46pc6\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.812447 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.812434 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.812488 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.812470 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.913203 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.913171 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.913395 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.913238 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46pc6\" (UniqueName: \"kubernetes.io/projected/b0c97856-79d6-4471-bcf8-36b0bc4e5037-kube-api-access-46pc6\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.913395 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.913310 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.913395 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.913338 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.913395 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.913380 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.913629 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.913439 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.913629 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.913473 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.913629 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.913521 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.913629 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.913559 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.913830 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.913730 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.914291 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.914261 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.914446 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.914312 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.914446 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.914381 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.914446 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.914399 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.915687 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.915663 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.915913 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.915889 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.951351 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.951317 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:34.951492 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:34.951370 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46pc6\" (UniqueName: \"kubernetes.io/projected/b0c97856-79d6-4471-bcf8-36b0bc4e5037-kube-api-access-46pc6\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:35.032535 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:35.032431 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:35.364144 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:35.364111 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh"] Oct 30 13:06:35.364878 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:06:35.364851 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c97856_79d6_4471_bcf8_36b0bc4e5037.slice/crio-2d3698a8256b04e1869b6e2e59ff2df48febad7b47c0686964becf275925f931 WatchSource:0}: Error finding container 2d3698a8256b04e1869b6e2e59ff2df48febad7b47c0686964becf275925f931: Status 404 returned error can't find the container with id 2d3698a8256b04e1869b6e2e59ff2df48febad7b47c0686964becf275925f931 Oct 30 13:06:35.367284 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:35.367247 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:35.367395 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:35.367334 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:35.367395 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:35.367377 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:35.607633 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:35.607587 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" event={"ID":"b0c97856-79d6-4471-bcf8-36b0bc4e5037","Type":"ContainerStarted","Data":"a042f802ce03ee9599b63629ae52c25e2828c9d58f4dde53567666f82ce0e156"} Oct 30 13:06:35.607810 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:35.607656 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" event={"ID":"b0c97856-79d6-4471-bcf8-36b0bc4e5037","Type":"ContainerStarted","Data":"2d3698a8256b04e1869b6e2e59ff2df48febad7b47c0686964becf275925f931"} Oct 30 13:06:35.654628 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:35.654562 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" podStartSLOduration=1.654540305 podStartE2EDuration="1.654540305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:35.653304808 +0000 UTC m=+889.476533098" watchObservedRunningTime="2026-10-30 13:06:35.654540305 +0000 UTC m=+889.477768583" Oct 30 13:06:36.033697 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:36.033601 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:36.035632 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:36.035604 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 13:06:36.035754 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:36.035652 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" podUID="b0c97856-79d6-4471-bcf8-36b0bc4e5037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 13:06:37.032928 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.032890 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 13:06:37.033124 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.032978 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" podUID="b0c97856-79d6-4471-bcf8-36b0bc4e5037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 13:06:37.106435 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.106405 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl"] Oct 30 13:06:37.110872 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.110847 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.141435 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.140765 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30750a5b-e095-46ec-bb64-fbfc4bfa7427-tls-certs\") pod \"distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.141435 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.140854 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-home\") pod \"distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.141435 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.140894 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-tmp-dir\") pod \"distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.141435 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.141004 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjb7l\" (UniqueName: \"kubernetes.io/projected/30750a5b-e095-46ec-bb64-fbfc4bfa7427-kube-api-access-qjb7l\") pod \"distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.141435 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.141049 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-model-cache\") pod \"distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.141435 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.141123 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-dshm\") pod \"distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.143130 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.143105 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb71e2\"/\"distinct-2-w2-bb71e2-kserve-self-signed-certs\"" Oct 30 13:06:37.196031 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.195992 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl"] Oct 30 13:06:37.241966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.241899 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-home\") pod \"distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.242163 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.241978 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-tmp-dir\") pod \"distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.242163 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.242119 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjb7l\" (UniqueName: \"kubernetes.io/projected/30750a5b-e095-46ec-bb64-fbfc4bfa7427-kube-api-access-qjb7l\") pod \"distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.242163 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.242156 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-model-cache\") pod \"distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.242327 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.242237 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-dshm\") pod \"distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.242382 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.242324 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30750a5b-e095-46ec-bb64-fbfc4bfa7427-tls-certs\") pod \"distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.242382 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.242372 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-home\") pod \"distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.242488 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.242423 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-tmp-dir\") pod \"distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.242588 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.242564 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-model-cache\") pod \"distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.255281 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.255248 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30750a5b-e095-46ec-bb64-fbfc4bfa7427-tls-certs\") pod \"distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.288797 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.288672 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-dshm\") pod \"distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.291618 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.291573 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjb7l\" (UniqueName: \"kubernetes.io/projected/30750a5b-e095-46ec-bb64-fbfc4bfa7427-kube-api-access-qjb7l\") pod \"distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.431016 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.430195 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:37.841514 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:37.841487 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl"] Oct 30 13:06:37.842635 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:06:37.842609 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30750a5b_e095_46ec_bb64_fbfc4bfa7427.slice/crio-ac3b41e6630e0d63b34cfbaf1325c68da10a0148d69132ab3087cf87180c31d1 WatchSource:0}: Error finding container ac3b41e6630e0d63b34cfbaf1325c68da10a0148d69132ab3087cf87180c31d1: Status 404 returned error can't find the container with id ac3b41e6630e0d63b34cfbaf1325c68da10a0148d69132ab3087cf87180c31d1 Oct 30 13:06:38.033533 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:38.033488 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 13:06:38.033727 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:38.033551 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" podUID="b0c97856-79d6-4471-bcf8-36b0bc4e5037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 13:06:38.642999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:38.642955 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" event={"ID":"30750a5b-e095-46ec-bb64-fbfc4bfa7427","Type":"ContainerStarted","Data":"7c5bf279ce61edc176b59830a165f458bdf4ef3f0cd6cd47d7acce13aa6a27e7"} Oct 30 13:06:38.643436 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:38.643011 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" event={"ID":"30750a5b-e095-46ec-bb64-fbfc4bfa7427","Type":"ContainerStarted","Data":"ac3b41e6630e0d63b34cfbaf1325c68da10a0148d69132ab3087cf87180c31d1"} Oct 30 13:06:38.697187 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:38.697024 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podStartSLOduration=1.697004294 podStartE2EDuration="1.697004294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:38.69203661 +0000 UTC m=+892.515264889" watchObservedRunningTime="2026-10-30 13:06:38.697004294 +0000 UTC m=+892.520232572" Oct 30 13:06:39.033121 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:39.033036 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 13:06:39.033121 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:39.033092 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" podUID="b0c97856-79d6-4471-bcf8-36b0bc4e5037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 13:06:39.431495 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:39.431451 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:40.033327 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:40.033290 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 13:06:40.033733 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:40.033348 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" podUID="b0c97856-79d6-4471-bcf8-36b0bc4e5037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 13:06:41.033745 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:41.033710 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 13:06:41.034146 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:41.033763 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" podUID="b0c97856-79d6-4471-bcf8-36b0bc4e5037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 13:06:42.033452 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:42.033410 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 13:06:42.033668 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:42.033471 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" podUID="b0c97856-79d6-4471-bcf8-36b0bc4e5037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 13:06:43.124120 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:43.124084 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:43.125115 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:43.125088 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:43.125906 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:43.125884 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:06:47.431581 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:47.431534 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:47.444137 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:47.444107 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:47.787399 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:47.787307 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:06:49.381970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.381912 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw"] Oct 30 13:06:49.388209 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.388177 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.407397 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.407366 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw"] Oct 30 13:06:49.509359 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.509310 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.509515 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.509381 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.509515 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.509480 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.509649 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.509617 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.509726 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.509706 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c4c258-7da6-40a5-82f5-618f8bea345f-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.509791 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.509779 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.509885 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.509865 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.509980 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.509917 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhzxt\" (UniqueName: \"kubernetes.io/projected/72c4c258-7da6-40a5-82f5-618f8bea345f-kube-api-access-hhzxt\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.510036 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.510010 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.611803 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.611764 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.611988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.611830 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c4c258-7da6-40a5-82f5-618f8bea345f-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.611988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.611856 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.611988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.611871 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.611988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.611887 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhzxt\" (UniqueName: \"kubernetes.io/projected/72c4c258-7da6-40a5-82f5-618f8bea345f-kube-api-access-hhzxt\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.611988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.611951 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.612296 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.612038 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.612296 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.612086 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.612296 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.612129 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.612445 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.612346 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.612445 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.612426 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.612749 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.612725 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.612829 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.612778 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.613113 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.613085 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c4c258-7da6-40a5-82f5-618f8bea345f-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.614584 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.614565 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.615031 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.615010 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.640612 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.640544 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.640775 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.640748 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhzxt\" (UniqueName: \"kubernetes.io/projected/72c4c258-7da6-40a5-82f5-618f8bea345f-kube-api-access-hhzxt\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:49.718050 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:49.718006 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:50.045817 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:50.045782 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw"] Oct 30 13:06:50.047464 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:06:50.047431 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c4c258_7da6_40a5_82f5_618f8bea345f.slice/crio-21ca8323d66d6b836a62957b3ec847a781a978835137c43ea376baca4b6e984a WatchSource:0}: Error finding container 21ca8323d66d6b836a62957b3ec847a781a978835137c43ea376baca4b6e984a: Status 404 returned error can't find the container with id 21ca8323d66d6b836a62957b3ec847a781a978835137c43ea376baca4b6e984a Oct 30 13:06:50.050579 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:50.050537 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:50.050685 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:50.050609 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:50.050685 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:50.050639 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:50.803462 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:50.803410 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" event={"ID":"72c4c258-7da6-40a5-82f5-618f8bea345f","Type":"ContainerStarted","Data":"8f2a566a128b7ac0b05681b0d74edb405d97067f4bc81d9dd0172351fd4cbacd"} Oct 30 13:06:50.804055 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:50.803539 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" event={"ID":"72c4c258-7da6-40a5-82f5-618f8bea345f","Type":"ContainerStarted","Data":"21ca8323d66d6b836a62957b3ec847a781a978835137c43ea376baca4b6e984a"} Oct 30 13:06:50.882895 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:50.882823 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" podStartSLOduration=1.8827868300000001 podStartE2EDuration="1.88278683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:50.8652202 +0000 UTC m=+904.688448478" watchObservedRunningTime="2026-10-30 13:06:50.88278683 +0000 UTC m=+904.706015108" Oct 30 13:06:51.719227 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:51.719190 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:06:51.721051 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:51.721023 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 13:06:51.721162 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:51.721073 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" podUID="72c4c258-7da6-40a5-82f5-618f8bea345f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 13:06:52.719344 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:52.719311 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 13:06:52.719847 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:52.719358 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" podUID="72c4c258-7da6-40a5-82f5-618f8bea345f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 13:06:53.719003 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:53.718967 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 13:06:53.719220 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:53.719020 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" podUID="72c4c258-7da6-40a5-82f5-618f8bea345f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 13:06:54.718705 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:54.718653 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 13:06:54.719156 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:54.718718 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" podUID="72c4c258-7da6-40a5-82f5-618f8bea345f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 13:06:55.718821 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:55.718785 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 13:06:55.719385 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:55.718847 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" podUID="72c4c258-7da6-40a5-82f5-618f8bea345f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 13:06:56.718359 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:56.718323 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 13:06:56.718668 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:56.718627 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" podUID="72c4c258-7da6-40a5-82f5-618f8bea345f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 13:06:58.768444 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:58.768403 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:06:58.769498 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:58.769190 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" podUID="72c4c258-7da6-40a5-82f5-618f8bea345f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:58.829175 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:58.829126 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:58.924094 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:58.922156 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:06:58.930970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:58.924375 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" podUID="c715c318-f465-4b5a-8c65-10c6539a5c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:59.646963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:59.646088 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:06:59.646963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:59.646224 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" podUID="88155745-668a-4d1c-8114-2c9eb2a13dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:59.712093 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:59.712045 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:59.743155 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:59.743114 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:06:59.743447 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:59.743420 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" podUID="3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:59.790118 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:59.790070 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:06:59.790734 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:06:59.790685 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" podUID="72c4c258-7da6-40a5-82f5-618f8bea345f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:06:59.968511 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:06:59.968382 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:07:00.141708 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:00.141665 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 13:07:00.187142 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:00.187103 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:00.199215 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:00.199172 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" podUID="b8e7baf7-ef26-41d4-9dec-f4987791862d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:00.792617 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:00.792571 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:00.793150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:00.793117 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" podUID="72c4c258-7da6-40a5-82f5-618f8bea345f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:01.839143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:01.839105 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:01.839965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:01.839905 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" podUID="72c4c258-7da6-40a5-82f5-618f8bea345f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:01.972743 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:01.972710 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:07:01.973186 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:01.973161 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:07:01.974068 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:01.974047 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:07:19.466151 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.466114 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d"] Oct 30 13:07:19.470628 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.470605 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.493977 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.493914 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.494191 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.493980 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.494191 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.494078 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.494191 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.494115 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.494353 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.494282 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7wtv\" (UniqueName: \"kubernetes.io/projected/c1d0fc5e-a51f-490d-892a-809a3464c0b6-kube-api-access-z7wtv\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.494353 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.494331 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.494450 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.494394 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.494498 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.494461 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.494545 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.494497 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.505176 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.505143 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d"] Oct 30 13:07:19.595570 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.595530 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.595770 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.595600 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.595770 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.595635 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.595770 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.595717 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.595770 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.595756 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.596013 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.595879 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7wtv\" (UniqueName: \"kubernetes.io/projected/c1d0fc5e-a51f-490d-892a-809a3464c0b6-kube-api-access-z7wtv\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.596013 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.595934 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.596013 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.596002 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.596170 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.596056 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.596170 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.596083 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-workload-certs\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.596288 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.596221 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-credential-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.596390 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.596367 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-workload-socket\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.596545 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.596414 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-data\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.596545 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.596465 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istiod-ca-cert\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.598194 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.598170 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-envoy\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.598485 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.598467 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-podinfo\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.621479 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.621453 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-token\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.621616 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.621475 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7wtv\" (UniqueName: \"kubernetes.io/projected/c1d0fc5e-a51f-490d-892a-809a3464c0b6-kube-api-access-z7wtv\") pod \"e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.789999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.789893 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:19.870934 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.870906 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd"] Oct 30 13:07:19.876825 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.876798 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:19.901473 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.900349 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:19.901473 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.900398 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:19.901473 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.900434 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:19.901473 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.900505 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:19.901473 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.900557 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:19.901473 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.900645 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:19.901473 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.900716 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:19.901473 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.900788 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmkdx\" (UniqueName: \"kubernetes.io/projected/cd5b17d2-dd01-422f-ade1-6b34aebe0048-kube-api-access-nmkdx\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:19.901473 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.900834 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:19.904910 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.904885 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd"] Oct 30 13:07:19.913728 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.913700 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf"] Oct 30 13:07:19.919585 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.919194 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:19.940833 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:19.940801 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf"] Oct 30 13:07:20.001727 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.001686 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.001727 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.001730 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njns9\" (UniqueName: \"kubernetes.io/projected/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-kube-api-access-njns9\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.002007 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.001754 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:20.002007 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.001824 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.002007 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.001930 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:20.002007 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.001995 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:20.002212 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.002021 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.002212 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.002083 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:20.002212 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.002109 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.002212 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.002194 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.002426 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.002239 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.002426 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.002324 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:20.002426 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.002369 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:20.002426 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.002412 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.002600 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.002457 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:20.002600 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.002508 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:20.002600 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.002545 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.002766 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.002624 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:20.002766 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.002722 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmkdx\" (UniqueName: \"kubernetes.io/projected/cd5b17d2-dd01-422f-ade1-6b34aebe0048-kube-api-access-nmkdx\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:20.002878 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.002774 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:20.003074 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.003042 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:20.003209 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.003183 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:20.003277 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.003225 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:20.004380 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.004357 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:20.004653 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.004631 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:20.030254 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.030223 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:20.030394 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.030290 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmkdx\" (UniqueName: \"kubernetes.io/projected/cd5b17d2-dd01-422f-ade1-6b34aebe0048-kube-api-access-nmkdx\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:20.104652 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.104556 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.104652 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.104605 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njns9\" (UniqueName: \"kubernetes.io/projected/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-kube-api-access-njns9\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.104652 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.104624 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.104652 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.104652 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.105024 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.104702 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.105024 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.104750 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.105024 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.104779 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.105024 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.104971 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.105268 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.105184 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-workload-certs\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.105331 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.105272 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-credential-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.105331 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.105293 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-data\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.105491 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.105471 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.105549 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.105515 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istiod-ca-cert\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.105585 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.105547 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-workload-socket\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.107109 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.107090 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-envoy\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.107321 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.107304 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-podinfo\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.115952 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.115910 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d"] Oct 30 13:07:20.116097 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:07:20.116077 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d0fc5e_a51f_490d_892a_809a3464c0b6.slice/crio-3c4dba5be3f220b2ba3bd77026d88c8f610a4bc8569e7f7adfc02b259424aed2 WatchSource:0}: Error finding container 3c4dba5be3f220b2ba3bd77026d88c8f610a4bc8569e7f7adfc02b259424aed2: Status 404 returned error can't find the container with id 3c4dba5be3f220b2ba3bd77026d88c8f610a4bc8569e7f7adfc02b259424aed2 Oct 30 13:07:20.118588 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.118558 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:20.118690 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.118635 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:20.118690 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.118672 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:20.130296 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.130263 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njns9\" (UniqueName: \"kubernetes.io/projected/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-kube-api-access-njns9\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.169988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.168958 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-token\") pod \"e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.203483 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.203222 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:20.209958 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.206118 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" event={"ID":"c1d0fc5e-a51f-490d-892a-809a3464c0b6","Type":"ContainerStarted","Data":"3c4dba5be3f220b2ba3bd77026d88c8f610a4bc8569e7f7adfc02b259424aed2"} Oct 30 13:07:20.244543 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.244510 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:20.420247 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.420214 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb"] Oct 30 13:07:20.425708 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.425675 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.436234 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.436197 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g"] Oct 30 13:07:20.460185 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.460147 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb"] Oct 30 13:07:20.460341 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.460279 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.511621 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.511588 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g"] Oct 30 13:07:20.515026 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.514997 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.515269 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.515239 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.515380 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.515285 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.515380 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.515314 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.515495 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.515374 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.515495 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.515425 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.515495 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.515470 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.515697 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.515558 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.515697 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.515611 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rpln\" (UniqueName: \"kubernetes.io/projected/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-kube-api-access-4rpln\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.515697 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.515658 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.515697 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.515686 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.515891 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.515770 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcvnh\" (UniqueName: \"kubernetes.io/projected/41e428c4-45aa-4d5d-b6ea-7939b76b2831-kube-api-access-xcvnh\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.515983 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.515919 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.516069 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.516052 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.516211 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.516173 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.516330 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.516313 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.516409 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.516390 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.516461 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.516418 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.604506 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.604483 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd"] Oct 30 13:07:20.607403 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.606307 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf"] Oct 30 13:07:20.607403 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.607009 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:20.607403 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.607077 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:20.607403 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.607106 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:20.607403 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.607157 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:20.607403 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.607224 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:20.607403 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.607272 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:20.617494 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.617463 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcvnh\" (UniqueName: \"kubernetes.io/projected/41e428c4-45aa-4d5d-b6ea-7939b76b2831-kube-api-access-xcvnh\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.617610 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.617541 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.617686 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.617615 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.617686 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.617653 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.617686 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.617679 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.617893 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.617722 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.617893 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.617745 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.617893 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.617779 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.617893 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.617814 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.617893 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.617851 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.617893 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.617883 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.618250 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.617997 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.618250 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.618039 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.618250 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.618083 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.618250 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.618124 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.618250 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.618164 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rpln\" (UniqueName: \"kubernetes.io/projected/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-kube-api-access-4rpln\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.618250 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.618212 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.618250 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.618236 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.618620 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.618542 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.618620 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.618571 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.618754 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.618675 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istiod-ca-cert\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.618907 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.618884 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.618988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.618968 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.621644 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.621619 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.621752 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.621709 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-envoy\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.621903 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.621882 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-credential-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.622031 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.621915 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.622135 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.622117 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.622263 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.622239 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-workload-certs\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.622343 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.622318 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-data\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.622798 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.622774 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-workload-socket\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.624517 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.624491 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-podinfo\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.646446 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.646368 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.648241 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.648212 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-token\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.651779 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.651755 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcvnh\" (UniqueName: \"kubernetes.io/projected/41e428c4-45aa-4d5d-b6ea-7939b76b2831-kube-api-access-xcvnh\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:20.652819 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.652790 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rpln\" (UniqueName: \"kubernetes.io/projected/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-kube-api-access-4rpln\") pod \"e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.764497 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.764423 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:20.797985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:20.791209 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:21.215748 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.215041 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:21.215748 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.215125 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:21.215748 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.215163 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:21.227861 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.227551 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb"] Oct 30 13:07:21.294227 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.294140 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" event={"ID":"2c8e76b7-c2ba-4dd2-8b49-2d4565580919","Type":"ContainerStarted","Data":"2c924ac93ea0395ca4265e7adfac188715b42b9f3d065c3c0f97ae508c7fc3b7"} Oct 30 13:07:21.298573 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.297674 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" event={"ID":"c1d0fc5e-a51f-490d-892a-809a3464c0b6","Type":"ContainerStarted","Data":"2c5cd52c2f885a5902ec0a75c2449ae37a2f8bafc649f5c60585f9ee3c1c923d"} Oct 30 13:07:21.301212 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.300428 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" event={"ID":"cd5b17d2-dd01-422f-ade1-6b34aebe0048","Type":"ContainerStarted","Data":"773d2cdaa980c929632b794e68d5c8a082faa0bf8b6998edbcd09404014067ad"} Oct 30 13:07:21.301212 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.300479 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" event={"ID":"cd5b17d2-dd01-422f-ade1-6b34aebe0048","Type":"ContainerStarted","Data":"b847034b4f2507c474d7a0357fee909a513e012c69a16b73bb1e47eb926402e1"} Oct 30 13:07:21.303627 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.303300 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" event={"ID":"79b4bb1d-abcd-4bda-98ed-e62781d1abd7","Type":"ContainerStarted","Data":"4f7a3c1e074e282189a0ef57f60d443a6333f93e6d727d30137ff638e908f6d7"} Oct 30 13:07:21.303627 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.303343 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" event={"ID":"79b4bb1d-abcd-4bda-98ed-e62781d1abd7","Type":"ContainerStarted","Data":"d6a2ba4d9ae7f214d20da67d34ea1afef0c3e4edf5b1c47a5061bab6a9d65923"} Oct 30 13:07:21.312425 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.312377 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g"] Oct 30 13:07:21.313494 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:07:21.313350 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e428c4_45aa_4d5d_b6ea_7939b76b2831.slice/crio-9d81b224a3ec9efa287db71ff5543e019fe799d69ea00728169762621d7da1ab WatchSource:0}: Error finding container 9d81b224a3ec9efa287db71ff5543e019fe799d69ea00728169762621d7da1ab: Status 404 returned error can't find the container with id 9d81b224a3ec9efa287db71ff5543e019fe799d69ea00728169762621d7da1ab Oct 30 13:07:21.316586 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.316558 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:21.316775 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.316759 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:21.316876 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.316865 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:21.403145 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.403075 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" podStartSLOduration=2.403052299 podStartE2EDuration="2.403052299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:21.344886853 +0000 UTC m=+935.168115131" watchObservedRunningTime="2026-10-30 13:07:21.403052299 +0000 UTC m=+935.226280577" Oct 30 13:07:21.409870 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.409788 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podStartSLOduration=2.409766912 podStartE2EDuration="2.409766912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:21.399414881 +0000 UTC m=+935.222643159" watchObservedRunningTime="2026-10-30 13:07:21.409766912 +0000 UTC m=+935.232995188" Oct 30 13:07:21.483021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.479170 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podStartSLOduration=2.479148002 podStartE2EDuration="2.479148002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:21.453576901 +0000 UTC m=+935.276805179" watchObservedRunningTime="2026-10-30 13:07:21.479148002 +0000 UTC m=+935.302376281" Oct 30 13:07:21.795463 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.795330 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:21.798768 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.798737 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 13:07:21.820099 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:21.820054 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" podUID="c1d0fc5e-a51f-490d-892a-809a3464c0b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 13:07:22.464083 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:22.418363 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:22.464083 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:22.448390 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:22.490989 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:22.480617 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:07:22.490989 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:22.480712 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 13:07:22.492540 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:22.492507 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:07:22.492771 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:22.492739 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 13:07:22.814136 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:22.814100 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 13:07:22.814763 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:22.814714 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" podUID="c1d0fc5e-a51f-490d-892a-809a3464c0b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 13:07:22.866143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:22.866081 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:23.108476 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:23.108380 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" event={"ID":"2c8e76b7-c2ba-4dd2-8b49-2d4565580919","Type":"ContainerStarted","Data":"950cdbe3dbb332f000742d6b98c34775166e3efde6b0e30622cbb12a2e0eb3eb"} Oct 30 13:07:23.138054 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:23.116964 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" event={"ID":"41e428c4-45aa-4d5d-b6ea-7939b76b2831","Type":"ContainerStarted","Data":"dbff5914e675f231f498e2360270d32d3712e2debf16db9d305e229a1ce35845"} Oct 30 13:07:23.249968 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:23.236377 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" event={"ID":"41e428c4-45aa-4d5d-b6ea-7939b76b2831","Type":"ContainerStarted","Data":"9d81b224a3ec9efa287db71ff5543e019fe799d69ea00728169762621d7da1ab"} Oct 30 13:07:23.249968 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:23.223030 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:07:23.249968 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:23.236486 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 13:07:23.249968 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:23.147040 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:23.251420 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:23.251356 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" podStartSLOduration=3.2513360159999998 podStartE2EDuration="3.251336016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:23.174414432 +0000 UTC m=+936.997642710" watchObservedRunningTime="2026-10-30 13:07:23.251336016 +0000 UTC m=+937.074564296" Oct 30 13:07:23.265899 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:23.264098 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:07:23.265899 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:23.264185 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 13:07:23.345965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:23.338804 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" podStartSLOduration=3.338779011 podStartE2EDuration="3.338779011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:23.266387979 +0000 UTC m=+937.089616289" watchObservedRunningTime="2026-10-30 13:07:23.338779011 +0000 UTC m=+937.162007293" Oct 30 13:07:23.715965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:23.715022 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:23.777176 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:23.777142 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:23.777548 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:23.777521 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:07:23.777748 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:23.777724 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 13:07:24.030680 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.030582 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 13:07:24.031265 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.031235 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" podUID="c1d0fc5e-a51f-490d-892a-809a3464c0b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 13:07:24.047810 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.047774 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:24.048219 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.048192 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:07:24.048409 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.048383 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" podUID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 13:07:24.074117 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.074066 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:24.074839 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.074808 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:07:24.269964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.269070 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:07:24.269964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.269132 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 13:07:24.270223 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.269999 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 13:07:24.283017 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.281053 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:07:24.598200 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.598099 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:07:24.598481 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.598447 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 13:07:24.729963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.729803 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:24.755482 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.739034 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:07:24.765293 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.765258 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:07:24.765552 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.765525 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 13:07:24.821135 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.821087 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 13:07:24.928965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.919779 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" podUID="c1d0fc5e-a51f-490d-892a-809a3464c0b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 13:07:24.934965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.843030 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:07:24.964174 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:24.964127 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" podUID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 13:07:25.076080 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:25.076031 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 13:07:25.225015 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:25.222597 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:07:25.225015 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:25.222656 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 13:07:25.272963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:25.256256 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:07:25.272963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:25.256320 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 13:07:25.281993 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:25.274865 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:25.533025 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:25.532537 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:07:25.757898 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:25.757862 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:07:25.794317 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:25.794228 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:07:25.794570 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:25.794542 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 13:07:25.795112 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:25.795089 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 13:07:25.795258 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:25.795239 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" podUID="c1d0fc5e-a51f-490d-892a-809a3464c0b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 13:07:25.803349 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:25.803317 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:07:25.803586 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:25.803549 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" podUID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 13:07:26.098029 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:26.084216 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:07:26.304782 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:26.304743 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:07:26.305094 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:26.305064 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 13:07:26.388680 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:26.382039 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:07:26.389252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:26.389210 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 13:07:26.782601 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:26.781134 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:07:26.782601 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:26.781215 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 13:07:26.801111 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:26.800697 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:26.817966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:26.811765 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:07:26.817966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:26.811843 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" podUID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 13:07:26.817966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:26.812018 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 13:07:26.817966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:26.812097 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" podUID="c1d0fc5e-a51f-490d-892a-809a3464c0b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 13:07:27.125029 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:27.112329 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:27.224988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:27.215411 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:07:27.224988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:27.215472 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 13:07:27.307977 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:27.306985 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:07:27.307977 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:27.307062 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 13:07:27.586525 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:27.586483 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:27.780787 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:27.780751 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:07:27.781087 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:27.781056 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 13:07:27.804583 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:27.804549 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 13:07:27.805215 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:27.805192 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" podUID="c1d0fc5e-a51f-490d-892a-809a3464c0b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 13:07:27.805368 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:27.805134 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:27.805735 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:27.805090 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:07:27.806611 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:27.806586 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" podUID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 13:07:27.806758 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:27.806343 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:07:28.152682 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:28.152448 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:28.153215 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:28.153166 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:07:28.223008 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:28.218071 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:07:28.223008 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:28.218174 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 13:07:28.280028 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:28.278088 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:07:28.280028 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:28.278151 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 13:07:28.596090 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:28.595989 4503 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:28.618128 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:28.616905 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:28.660965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:28.642865 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:07:28.824404 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:28.823075 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:28.876038 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:28.849035 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:07:28.876038 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:28.849101 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 13:07:28.891698 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:28.891656 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:07:28.892048 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:28.892015 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" podUID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 13:07:29.110963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:29.088051 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 13:07:29.110963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:29.088119 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" podUID="c1d0fc5e-a51f-490d-892a-809a3464c0b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 13:07:29.177966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:29.170449 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:29.242962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:29.240115 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:07:29.242962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:29.240187 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 13:07:29.271709 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:29.271659 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:07:29.272002 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:29.271969 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 13:07:29.618377 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:29.618268 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:29.770029 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:29.765350 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:07:29.770029 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:29.765416 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 13:07:29.794966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:29.790321 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 13:07:29.794966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:29.790388 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" podUID="c1d0fc5e-a51f-490d-892a-809a3464c0b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 13:07:29.807962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:29.798326 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:07:29.807962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:29.798387 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" podUID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 13:07:29.823617 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:29.823576 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 13:07:30.071977 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:07:30.068498 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:07:30.187963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:30.172020 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:30.224951 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:30.224451 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:07:30.224951 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:30.224529 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 13:07:30.250512 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:30.250477 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:07:30.250786 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:30.250758 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 13:07:30.591163 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:30.591097 4503 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:30.625319 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:30.625277 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:30.770965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:30.770458 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:07:30.770965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:30.770541 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 13:07:30.803818 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:30.803767 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:07:30.805603 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:30.805571 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" podUID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 13:07:30.812338 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:30.812308 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 13:07:30.812588 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:30.812559 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" podUID="c1d0fc5e-a51f-490d-892a-809a3464c0b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 13:07:30.898032 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:30.897066 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:30.934521 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:30.934474 4503 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:31.172558 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:31.172459 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:31.204263 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:31.204224 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:07:31.204824 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:31.204793 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 13:07:31.247889 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:31.247774 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:07:31.247889 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:31.247830 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 13:07:31.640034 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:31.639904 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:31.767973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:31.765388 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:07:31.767973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:31.765460 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 13:07:31.802489 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:31.802453 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:07:31.802791 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:31.802764 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" podUID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 13:07:31.803644 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:31.803617 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 13:07:31.803817 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:31.803796 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" podUID="c1d0fc5e-a51f-490d-892a-809a3464c0b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 13:07:31.833880 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:31.833849 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:07:32.183111 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:32.183050 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:32.265967 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:32.237166 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:07:32.265967 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:32.237229 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 13:07:32.269961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:32.269458 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 13:07:32.303963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:32.279914 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 13:07:32.622516 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:32.622405 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:07:32.804073 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:32.804024 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:07:32.804348 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:32.804319 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 13:07:32.805555 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:32.805526 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:07:32.806784 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:32.806758 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" podUID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 13:07:32.882709 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:32.882597 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:07:33.302110 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:33.243902 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 13:07:33.302110 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:33.244016 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 13:07:33.768073 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:33.768034 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 13:07:33.768350 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:33.768323 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 13:07:33.800962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:33.798404 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:07:33.800962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:33.798469 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" podUID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 13:07:33.861964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:33.861075 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:33.861964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:33.861150 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" podUID="c1d0fc5e-a51f-490d-892a-809a3464c0b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:34.397964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:34.367097 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:34.399113 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:34.399069 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:34.683023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:34.679122 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:34.806065 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:34.806020 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:34.829348 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:34.829310 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:34.830011 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:34.829902 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:35.176155 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:35.175021 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:35.213154 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:35.213122 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:35.254146 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:35.254101 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:35.274965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:35.272059 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:35.274965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:35.272151 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:35.446197 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:35.446085 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:35.446817 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:35.446767 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:35.770347 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:35.770240 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:35.778194 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:35.778160 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:07:35.779064 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:35.779037 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:35.779290 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:35.779243 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:35.797077 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:35.797039 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:35.836065 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:35.836024 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:35.836350 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:35.836317 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" podUID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:35.878966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:35.874047 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:35.878966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:35.874469 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:07:36.214962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:36.212254 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:36.214962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:36.212848 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:07:36.287009 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:36.286407 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:36.287009 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:36.286484 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:36.520985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:36.520877 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:36.521602 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:36.521567 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:36.584743 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:36.582334 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:07:36.584743 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:36.583537 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:07:36.683058 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:36.683017 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:36.809057 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:36.799919 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:07:36.938966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:36.931221 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:07:36.980963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:36.859397 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:37.040964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:37.015669 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:07:37.416986 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:37.336687 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:37.416986 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:37.336751 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:37.416986 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:37.396480 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:37.548321 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:37.548281 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:37.548932 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:37.548886 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:37.564712 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:37.564683 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:07:37.607963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:37.589019 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 13:07:38.382594 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:38.382543 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:38.382917 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:38.382868 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:38.604619 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:38.604574 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:38.605282 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:38.605244 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:38.792746 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:38.792707 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:38.794140 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:38.794108 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:07:38.820864 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:38.820821 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:38.822744 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:38.822716 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:07:53.672828 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:53.672764 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:54.633966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:54.602630 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:54.723080 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:54.689880 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:54.768245 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:54.767254 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:54.777045 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:54.768682 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:07:54.879750 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:54.879706 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:54.880053 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:54.880017 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:55.484964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:55.447114 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:55.484964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:55.447196 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:55.744023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:55.743299 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 13:07:55.753670 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:55.753636 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:07:55.776858 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:55.776815 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:55.780520 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:55.780480 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:55.781809 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:55.781786 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:07:56.296046 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:56.292537 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:07:56.296046 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:56.292621 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:07:56.296046 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:07:56.294999 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:07:59.702934 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:07:59.702891 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:08:06.333965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:06.327084 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 13:08:06.333965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:06.327642 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:06.333965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:06.329316 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:07.380214 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:07.380156 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 13:08:07.381192 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:07.381160 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 13:08:07.387963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:07.382056 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:08:07.387963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:07.382118 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:08:08.240428 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:08.240034 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:08.240428 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:08.240123 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" podUID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:08.321582 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:08.320493 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:08.395960 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:08.395894 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:08.424508 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:08.397142 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 13:08:08.442113 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:08.442065 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:08.442407 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:08.442371 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:08.489999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:08.484247 4503 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:08.676555 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:08.676518 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:08:09.469132 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:09.469088 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 13:08:09.514086 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:09.514036 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:09.552784 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:09.552750 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:08:09.865563 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:09.865460 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:09.865830 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:09.865797 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:10.218589 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:10.218555 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:08:10.531406 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:10.531251 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:08:21.430088 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:21.430031 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:21.610456 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:21.610406 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:22.442967 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:22.416909 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 13:08:22.495652 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:22.478369 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:22.559964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:22.547902 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:08:22.613119 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:22.613070 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:22.621964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:22.614089 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:08:23.065491 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:23.065456 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:23.065768 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:23.065740 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" podUID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:23.439627 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:23.437188 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:23.439627 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:23.437846 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:08:23.498549 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:23.498502 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:23.499188 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:23.499152 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:23.551645 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:23.505785 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:23.616966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:23.616023 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:24.470242 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:24.461802 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:24.566876 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:24.566824 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:24.617115 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:24.617068 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:24.899965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:24.897123 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:24.899965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:24.897194 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:24.939192 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:24.939151 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:24.939522 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:24.939470 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:25.462117 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:25.462063 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:25.575617 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:25.575556 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:25.619084 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:25.619037 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:25.758167 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:25.758071 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:08:26.644120 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:26.644071 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 13:08:26.648733 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:26.648699 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:26.785239 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:26.785203 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:08:26.846299 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:26.846264 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:08:29.703157 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:08:29.703116 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:08:42.592959 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:42.589065 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:43.358615 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:43.358574 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:43.362995 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:43.358908 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" podUID="25ebe375-c4db-41eb-bc87-3cb9d8e0853b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:43.593026 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:43.582224 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:43.628645 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:43.626271 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:43.940097 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:43.940052 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:43.947964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:43.944039 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" podUID="c715c318-f465-4b5a-8c65-10c6539a5c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:43.999627 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:43.999588 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:43.999888 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:43.999859 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" podUID="f263d119-9ab0-440f-ac86-fa5a0d2a2819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:44.598616 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:44.582563 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:44.609234 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:44.609142 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:44.626813 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:44.626758 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:44.645977 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:44.645049 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:44.645977 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:44.645132 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" podUID="88155745-668a-4d1c-8114-2c9eb2a13dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:44.675080 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:44.675039 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:44.675336 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:44.675312 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" podUID="51742836-262a-4345-ad56-bb8725b209ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:44.742990 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:44.740048 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:44.742990 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:44.740112 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" podUID="3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:44.864144 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:44.864050 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:44.864414 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:44.864386 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" podUID="70f0f890-bb6a-4627-9487-90741d34337c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:44.866448 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:44.866421 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:08:44.908255 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:44.908225 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:08:45.093107 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:45.093066 4503 patch_prober.go:29] interesting pod/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:45.093439 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:45.093398 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" podUID="b8e7baf7-ef26-41d4-9dec-f4987791862d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:45.637192 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:45.637146 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:45.806398 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:45.663057 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:45.810993 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:45.663134 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:45.841436 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:45.823246 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:08:46.842545 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:46.842407 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:46.844189 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:46.844149 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:47.084104 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:46.844900 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:47.865965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:47.865116 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:48.062961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:48.062373 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:48.062961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:48.062460 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" podUID="72c4c258-7da6-40a5-82f5-618f8bea345f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:48.087997 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:48.087897 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 13:08:48.088998 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:48.088592 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 13:08:48.440000 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:48.436159 4503 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:48.863025 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:48.859547 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:08:48.888964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:48.883985 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 13:08:48.943970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:48.942424 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:08:49.003173 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:08:49.003140 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:08:59.702689 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:08:59.702609 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:09:02.859706 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:02.859671 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8648-e2d0d0"] Oct 30 13:09:02.865076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:02.865049 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8648-e2d0d0" Oct 30 13:09:02.886977 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:02.885802 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8648-e2d0d0"] Oct 30 13:09:02.891277 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:02.891248 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4gdbp\"" Oct 30 13:09:02.956659 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:02.956402 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95zds\" (UniqueName: \"kubernetes.io/projected/976ca170-0822-440c-9e3e-e4598f72185a-kube-api-access-95zds\") pod \"test-curl-8648-e2d0d0\" (UID: \"976ca170-0822-440c-9e3e-e4598f72185a\") " pod="opendatahub/test-curl-8648-e2d0d0" Oct 30 13:09:03.057861 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:03.057821 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95zds\" (UniqueName: \"kubernetes.io/projected/976ca170-0822-440c-9e3e-e4598f72185a-kube-api-access-95zds\") pod \"test-curl-8648-e2d0d0\" (UID: \"976ca170-0822-440c-9e3e-e4598f72185a\") " pod="opendatahub/test-curl-8648-e2d0d0" Oct 30 13:09:03.082510 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:03.082476 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95zds\" (UniqueName: \"kubernetes.io/projected/976ca170-0822-440c-9e3e-e4598f72185a-kube-api-access-95zds\") pod \"test-curl-8648-e2d0d0\" (UID: \"976ca170-0822-440c-9e3e-e4598f72185a\") " pod="opendatahub/test-curl-8648-e2d0d0" Oct 30 13:09:03.181318 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:03.181281 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8648-e2d0d0" Oct 30 13:09:03.497351 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:03.497318 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8648-e2d0d0"] Oct 30 13:09:03.497832 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:09:03.497804 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod976ca170_0822_440c_9e3e_e4598f72185a.slice/crio-ddaab684bb22c85b28c49d5066d5597e07c524208915faeaa1a27e3733cb7d26 WatchSource:0}: Error finding container ddaab684bb22c85b28c49d5066d5597e07c524208915faeaa1a27e3733cb7d26: Status 404 returned error can't find the container with id ddaab684bb22c85b28c49d5066d5597e07c524208915faeaa1a27e3733cb7d26 Oct 30 13:09:03.499820 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:03.499802 4503 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:09:03.836760 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:03.836675 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8648-e2d0d0" event={"ID":"976ca170-0822-440c-9e3e-e4598f72185a","Type":"ContainerStarted","Data":"ddaab684bb22c85b28c49d5066d5597e07c524208915faeaa1a27e3733cb7d26"} Oct 30 13:09:05.451053 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:05.451006 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj"] Oct 30 13:09:05.778647 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:05.778563 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj"] Oct 30 13:09:05.778987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:05.778965 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:05.849963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:05.837996 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:05.849963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:05.838059 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4qnt\" (UniqueName: \"kubernetes.io/projected/4151e97e-c25a-49af-8268-baa1c493ee4a-kube-api-access-d4qnt\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:05.849963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:05.838123 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:05.849963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:05.838201 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:05.849963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:05.838237 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:05.849963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:05.838272 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4151e97e-c25a-49af-8268-baa1c493ee4a-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:05.849963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:05.838297 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:05.849963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:05.838443 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:05.849963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:05.838473 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.051030 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.041530 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.051030 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.041633 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.051030 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.041783 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.051030 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.041830 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4151e97e-c25a-49af-8268-baa1c493ee4a-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.051030 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.041861 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.076255 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.076076 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw"] Oct 30 13:09:06.163970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.160726 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.163970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.160799 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.163970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.161011 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.163970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.161054 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4qnt\" (UniqueName: \"kubernetes.io/projected/4151e97e-c25a-49af-8268-baa1c493ee4a-kube-api-access-d4qnt\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.163970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.162076 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.163970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.162362 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.188965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.170103 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.188965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.170842 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4151e97e-c25a-49af-8268-baa1c493ee4a-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.188965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.171156 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.378578 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.378481 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.418965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.401523 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5"] Oct 30 13:09:06.588982 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.581301 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.588982 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.581408 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.588982 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.581519 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e6b947-1515-4141-a00e-894c60bc30f1-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.588982 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.581730 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e6b947-1515-4141-a00e-894c60bc30f1-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.588982 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.581919 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.588982 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.581975 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e6b947-1515-4141-a00e-894c60bc30f1-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.588982 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.582146 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.588982 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.582301 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.588982 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.582344 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ck6l\" (UniqueName: \"kubernetes.io/projected/05e6b947-1515-4141-a00e-894c60bc30f1-kube-api-access-2ck6l\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.757953 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.757900 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:06.812998 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.788103 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4qnt\" (UniqueName: \"kubernetes.io/projected/4151e97e-c25a-49af-8268-baa1c493ee4a-kube-api-access-d4qnt\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.812998 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.789032 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.831023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.820391 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.831023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.820456 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e6b947-1515-4141-a00e-894c60bc30f1-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.831023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.820518 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.831023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.820560 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.831023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.820590 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ck6l\" (UniqueName: \"kubernetes.io/projected/05e6b947-1515-4141-a00e-894c60bc30f1-kube-api-access-2ck6l\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.831023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.820652 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.831023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.820719 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.831023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.820810 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e6b947-1515-4141-a00e-894c60bc30f1-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.831023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.820923 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e6b947-1515-4141-a00e-894c60bc30f1-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.877300 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.877164 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:06.883067 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.883023 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:06.909345 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.909307 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.909696 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.909674 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e6b947-1515-4141-a00e-894c60bc30f1-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.909889 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.909871 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.910393 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.910372 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.911011 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.910707 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.917843 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.917051 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.928702 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.920132 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:06.949454 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.949420 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:06.967625 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:06.967505 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw"] Oct 30 13:09:07.182281 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.182229 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56vch\" (UniqueName: \"kubernetes.io/projected/b366d790-421f-4e06-a5a9-5473df838ef5-kube-api-access-56vch\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.199961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.196782 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b366d790-421f-4e06-a5a9-5473df838ef5-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.199961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.196965 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.199961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.196998 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b366d790-421f-4e06-a5a9-5473df838ef5-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.199961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.197066 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.199961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.197235 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.199961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.197336 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.199961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.197373 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b366d790-421f-4e06-a5a9-5473df838ef5-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.199961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.197430 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.199961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.185209 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:07.304268 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.296546 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e6b947-1515-4141-a00e-894c60bc30f1-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:07.361026 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.335414 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.361743 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.361703 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.361978 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.361929 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b366d790-421f-4e06-a5a9-5473df838ef5-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.362177 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.362133 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.362316 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.362300 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56vch\" (UniqueName: \"kubernetes.io/projected/b366d790-421f-4e06-a5a9-5473df838ef5-kube-api-access-56vch\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.362526 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.362508 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b366d790-421f-4e06-a5a9-5473df838ef5-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.362767 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.362747 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.362898 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.362881 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b366d790-421f-4e06-a5a9-5473df838ef5-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.363089 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.363068 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.395966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.367780 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-istio-data\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.415961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.396624 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-workload-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.461963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.455354 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ck6l\" (UniqueName: \"kubernetes.io/projected/05e6b947-1515-4141-a00e-894c60bc30f1-kube-api-access-2ck6l\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:07.467918 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.467865 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-workload-certs\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.472311 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.472279 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-credential-socket\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.527773 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.526589 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b366d790-421f-4e06-a5a9-5473df838ef5-istiod-ca-cert\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.553963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.527698 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e6b947-1515-4141-a00e-894c60bc30f1-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:07.618838 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.618715 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56vch\" (UniqueName: \"kubernetes.io/projected/b366d790-421f-4e06-a5a9-5473df838ef5-kube-api-access-56vch\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.924408 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.893978 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5"] Oct 30 13:09:07.928125 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.928080 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:07.943505 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.943457 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:07.944288 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.944264 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:09:07.944486 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.944472 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:09:07.956170 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.956142 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b366d790-421f-4e06-a5a9-5473df838ef5-istio-token\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:07.961967 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:07.957298 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b366d790-421f-4e06-a5a9-5473df838ef5-istio-podinfo\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:08.256447 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:08.205391 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:08.256447 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:08.252478 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:09:08.678188 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:08.678148 4503 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded" Oct 30 13:09:08.771963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:08.765351 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:08.973655 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:08.797470 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-istio-envoy\") pod \"e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:08.973924 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:08.962801 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:08.979536 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:08.962827 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:09.077193 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:09.077149 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:09.320458 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:09.320355 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 13:09:09.383213 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:09.383177 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:09.954112 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:09.954072 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:09:09.954661 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:09.954633 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:10.041271 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:10.041224 4503 patch_prober.go:29] interesting pod/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:09:10.041557 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:10.041525 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:10.045051 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:10.045018 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 13:09:10.045573 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:10.045543 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:11.029254 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:10.947291 4503 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:11.231964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:11.195549 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:12.046023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:11.259116 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:12.092269 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:10.961500 4503 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded" Oct 30 13:09:12.092578 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:11.661337 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7tjtx" podUID="5522db7b-cfb1-4355-bbe0-b5022eac5e82" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 13:09:12.296946 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:12.277746 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 13:09:12.337444 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:09:12.337405 4503 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.76s" Oct 30 13:09:12.337804 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:12.337779 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j"] Oct 30 13:09:12.507021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:12.488588 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr" podUID="76757b33-71a1-4186-8ab0-b24bfb08d1c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:13.051246 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.051142 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:13.077979 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.060073 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:13.108399 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.108335 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:13.108600 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.108519 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:13.108664 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.108609 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:13.373873 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.373791 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 13:09:13.397972 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.392864 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.422348 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.422320 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j"] Oct 30 13:09:13.482995 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.463374 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-751d84fb-openshift-default-dockercfg-jspn4\"" Oct 30 13:09:13.488870 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.488834 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj"] Oct 30 13:09:13.530735 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.530598 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:09:13.531418 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.531399 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:09:13.540920 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.540886 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" event={"ID":"4151e97e-c25a-49af-8268-baa1c493ee4a","Type":"ContainerStarted","Data":"87aaee5368362c1985d0fcbd1de3734edbbea6580a117eee5df78ff621ed5874"} Oct 30 13:09:13.597959 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.559541 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5"] Oct 30 13:09:13.597959 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.569441 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw"] Oct 30 13:09:13.597959 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.581262 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:09:13.597959 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:09:13.583247 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb366d790_421f_4e06_a5a9_5473df838ef5.slice/crio-a03ce5565a872b7ad4182d2d1ceac40a427275e91a871f84896926f55419c68d WatchSource:0}: Error finding container a03ce5565a872b7ad4182d2d1ceac40a427275e91a871f84896926f55419c68d: Status 404 returned error can't find the container with id a03ce5565a872b7ad4182d2d1ceac40a427275e91a871f84896926f55419c68d Oct 30 13:09:13.597959 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.594611 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-credential-socket\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.597959 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.594655 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istiod-ca-cert\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.597959 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.594689 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-envoy\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.597959 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.594761 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr5c6\" (UniqueName: \"kubernetes.io/projected/1cc2d750-6f71-4f16-8228-82e6390d9bcf-kube-api-access-kr5c6\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.597959 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.594920 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-podinfo\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.597959 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.595076 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-workload-socket\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.597959 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.595257 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-workload-certs\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.597959 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.595375 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-token\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.597959 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.595457 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-data\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.608611 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.608564 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:13.608909 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.608886 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:13.609082 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.609064 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:13.628040 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.627059 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:13.628040 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.627147 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:13.628040 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.627188 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:13.702062 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.698515 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-workload-certs\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.702062 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.698558 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-token\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.702062 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.698614 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-data\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.702062 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.698688 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-credential-socket\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.702062 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.698714 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istiod-ca-cert\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.702062 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.698741 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-envoy\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.702062 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.698807 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr5c6\" (UniqueName: \"kubernetes.io/projected/1cc2d750-6f71-4f16-8228-82e6390d9bcf-kube-api-access-kr5c6\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.702062 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.698912 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-podinfo\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.702062 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.699029 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-workload-socket\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.702062 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.700245 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-workload-socket\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.702062 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.701084 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-data\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.702062 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.701355 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-credential-socket\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.702062 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.701899 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-envoy\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.702828 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.702458 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istiod-ca-cert\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.703076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.703025 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-workload-certs\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.710507 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.710477 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-podinfo\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.734121 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.734059 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr5c6\" (UniqueName: \"kubernetes.io/projected/1cc2d750-6f71-4f16-8228-82e6390d9bcf-kube-api-access-kr5c6\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.738693 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.738661 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-token\") pod \"e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:13.827447 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:13.827414 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:14.268798 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:14.268700 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j"] Oct 30 13:09:14.279055 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:09:14.279023 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc2d750_6f71_4f16_8228_82e6390d9bcf.slice/crio-b0041d79478b8330d1a988ca474707503b7d1ff20461210a4313e0854248f008 WatchSource:0}: Error finding container b0041d79478b8330d1a988ca474707503b7d1ff20461210a4313e0854248f008: Status 404 returned error can't find the container with id b0041d79478b8330d1a988ca474707503b7d1ff20461210a4313e0854248f008 Oct 30 13:09:14.282417 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:14.282377 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:14.282539 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:14.282476 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:14.282539 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:14.282512 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:14.554184 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:14.553804 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" event={"ID":"05e6b947-1515-4141-a00e-894c60bc30f1","Type":"ContainerStarted","Data":"9ce299a716a2721ee7e23efb1a658e127b5f1832db673418bc344bb1eddcbf4b"} Oct 30 13:09:14.554184 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:14.553895 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" event={"ID":"05e6b947-1515-4141-a00e-894c60bc30f1","Type":"ContainerStarted","Data":"b3ea7fcc17403feb4195f3617687b5e93768b3b0e6256f51edde0ca3a9564873"} Oct 30 13:09:14.560308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:14.560112 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" event={"ID":"1cc2d750-6f71-4f16-8228-82e6390d9bcf","Type":"ContainerStarted","Data":"a0bb50ffd774b138f89e420793c04af18f1c7fb10cd349438c0ced1c6eeb1c25"} Oct 30 13:09:14.560308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:14.560176 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" event={"ID":"1cc2d750-6f71-4f16-8228-82e6390d9bcf","Type":"ContainerStarted","Data":"b0041d79478b8330d1a988ca474707503b7d1ff20461210a4313e0854248f008"} Oct 30 13:09:14.565117 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:14.564951 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" event={"ID":"4151e97e-c25a-49af-8268-baa1c493ee4a","Type":"ContainerStarted","Data":"519b9a429ca3e6daf718e2c698e775b7b40f6e08bbcb820cd1ddbb85934e41ea"} Oct 30 13:09:14.575392 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:14.575258 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" event={"ID":"b366d790-421f-4e06-a5a9-5473df838ef5","Type":"ContainerStarted","Data":"86c7c0c2303b98548b7e6ceb7a46df90a2d0991bf8afdef66627778cba17574b"} Oct 30 13:09:14.575784 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:14.575746 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" event={"ID":"b366d790-421f-4e06-a5a9-5473df838ef5","Type":"ContainerStarted","Data":"a03ce5565a872b7ad4182d2d1ceac40a427275e91a871f84896926f55419c68d"} Oct 30 13:09:14.687882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:14.685777 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" podStartSLOduration=9.685753334 podStartE2EDuration="9.685753334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:14.670956544 +0000 UTC m=+1048.494184818" watchObservedRunningTime="2026-10-30 13:09:14.685753334 +0000 UTC m=+1048.508981613" Oct 30 13:09:14.687882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:14.686610 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" podStartSLOduration=9.686595021 podStartE2EDuration="9.686595021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:14.612882006 +0000 UTC m=+1048.436110284" watchObservedRunningTime="2026-10-30 13:09:14.686595021 +0000 UTC m=+1048.509823300" Oct 30 13:09:14.730963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:14.722585 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" podStartSLOduration=9.722565067 podStartE2EDuration="9.722565067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:14.720783375 +0000 UTC m=+1048.544011667" watchObservedRunningTime="2026-10-30 13:09:14.722565067 +0000 UTC m=+1048.545793346" Oct 30 13:09:14.865021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:14.846332 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" podStartSLOduration=3.846306941 podStartE2EDuration="3.846306941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:14.787987827 +0000 UTC m=+1048.611216104" watchObservedRunningTime="2026-10-30 13:09:14.846306941 +0000 UTC m=+1048.669535218" Oct 30 13:09:14.893367 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:14.855171 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:14.944093 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:14.944037 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:14.980641 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:14.979147 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:09:14.980641 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:14.979239 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" podUID="4151e97e-c25a-49af-8268-baa1c493ee4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 13:09:15.083185 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:15.083143 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:15.084752 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:15.084724 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 13:09:15.090089 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:15.090032 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" podUID="05e6b947-1515-4141-a00e-894c60bc30f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 13:09:15.402743 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:15.402710 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:15.403445 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:15.403421 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:09:15.403609 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:15.403572 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" podUID="b366d790-421f-4e06-a5a9-5473df838ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 13:09:15.857028 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:15.849339 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:15.923965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:15.923411 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:09:15.923965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:15.923489 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" podUID="4151e97e-c25a-49af-8268-baa1c493ee4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 13:09:16.089964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:16.089370 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 13:09:16.089964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:16.089432 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" podUID="05e6b947-1515-4141-a00e-894c60bc30f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 13:09:16.430365 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:16.424291 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:09:16.430977 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:16.430922 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" podUID="b366d790-421f-4e06-a5a9-5473df838ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 13:09:16.677245 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:16.677209 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:16.680039 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:16.680018 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:09:16.989740 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:16.989116 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:09:16.989740 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:16.989184 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" podUID="4151e97e-c25a-49af-8268-baa1c493ee4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 13:09:17.123995 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:17.115690 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 13:09:17.129071 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:17.124268 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" podUID="05e6b947-1515-4141-a00e-894c60bc30f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 13:09:17.649238 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:17.649202 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:09:17.736143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:17.736079 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" podUID="b366d790-421f-4e06-a5a9-5473df838ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 13:09:17.936665 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:17.936553 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:09:17.936985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:17.936891 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" podUID="4151e97e-c25a-49af-8268-baa1c493ee4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 13:09:18.090966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:18.081617 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 13:09:18.090966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:18.081687 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" podUID="05e6b947-1515-4141-a00e-894c60bc30f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 13:09:18.401877 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:18.401838 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:09:18.402242 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:18.402212 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" podUID="b366d790-421f-4e06-a5a9-5473df838ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 13:09:19.104974 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:19.086811 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:09:19.104974 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:19.086903 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" podUID="4151e97e-c25a-49af-8268-baa1c493ee4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 13:09:19.217295 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:19.215149 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 13:09:19.217295 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:19.215247 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" podUID="05e6b947-1515-4141-a00e-894c60bc30f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 13:09:19.465420 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:19.465381 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:09:19.465753 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:19.465721 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" podUID="b366d790-421f-4e06-a5a9-5473df838ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 13:09:19.945066 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:19.932539 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:09:19.945066 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:19.932601 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" podUID="4151e97e-c25a-49af-8268-baa1c493ee4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 13:09:20.098965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:20.098371 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 13:09:20.098965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:20.098451 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" podUID="05e6b947-1515-4141-a00e-894c60bc30f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 13:09:20.427963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:20.427174 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:09:20.427963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:20.427235 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" podUID="b366d790-421f-4e06-a5a9-5473df838ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 13:09:20.926358 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:20.925532 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:09:20.950442 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:20.925630 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" podUID="4151e97e-c25a-49af-8268-baa1c493ee4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 13:09:21.066478 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:21.066435 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 13:09:21.081849 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:21.081699 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:21.082260 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:21.082234 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 13:09:21.085955 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:21.082418 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" podUID="05e6b947-1515-4141-a00e-894c60bc30f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 13:09:21.414858 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:21.414420 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:09:21.414858 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:21.414482 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" podUID="b366d790-421f-4e06-a5a9-5473df838ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 13:09:21.922388 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:21.922345 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:21.923419 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:21.923387 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:09:21.923563 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:21.923438 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" podUID="4151e97e-c25a-49af-8268-baa1c493ee4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 13:09:22.080035 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:22.079674 4503 patch_prober.go:29] interesting pod/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 13:09:22.080035 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:22.079733 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" podUID="05e6b947-1515-4141-a00e-894c60bc30f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 13:09:23.107284 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:23.107252 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:23.109092 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:23.108982 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:23.111530 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:23.111478 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:09:23.223857 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:23.223741 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:23.224381 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:23.224350 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:23.225879 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:23.225854 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:09:23.730462 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:23.729774 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:23.732337 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:23.732160 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:09:24.924345 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:24.924311 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8648-e2d0d0" event={"ID":"976ca170-0822-440c-9e3e-e4598f72185a","Type":"ContainerStarted","Data":"feb071fef1e162fee004d1d65616db19eed7ee394ec91c093eb695ec2df7344f"} Oct 30 13:09:24.966256 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:24.966196 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8648-e2d0d0" podStartSLOduration=2.065348297 podStartE2EDuration="22.966179732s" totalImagesPullingTime="20.900831435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:24.964758658 +0000 UTC m=+1058.787986935" watchObservedRunningTime="2026-10-30 13:09:24.966179732 +0000 UTC m=+1058.789408054" Oct 30 13:09:25.963877 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:25.963838 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8648-e2d0d0"] Oct 30 13:09:25.974324 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:25.974285 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8648-e2d0d0"] Oct 30 13:09:26.938328 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:26.938285 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8648-e2d0d0" podUID="976ca170-0822-440c-9e3e-e4598f72185a" containerName="test-curl-8648-e2d0d0" containerID="cri-o://feb071fef1e162fee004d1d65616db19eed7ee394ec91c093eb695ec2df7344f" gracePeriod=2 Oct 30 13:09:27.666152 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:27.666116 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh"] Oct 30 13:09:27.668811 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:27.668773 4503 cpu_manager.go:414] "containerMap: removing container" podUID="976ca170-0822-440c-9e3e-e4598f72185a" containerName="test-curl-8648-e2d0d0" Oct 30 13:09:27.669482 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:27.669461 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="976ca170-0822-440c-9e3e-e4598f72185a" containerName="test-curl-8648-e2d0d0" Oct 30 13:09:27.670393 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:27.670374 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="976ca170-0822-440c-9e3e-e4598f72185a" containerName="test-curl-8648-e2d0d0" Oct 30 13:09:27.677781 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:27.677754 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh" Oct 30 13:09:27.680764 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:27.680721 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh"] Oct 30 13:09:27.733371 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:27.719320 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4a59cbc1\"" Oct 30 13:09:27.733371 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:27.719569 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4a59cbc1-dockercfg-8tx8j\"" Oct 30 13:09:27.811086 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:27.808244 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8-plugins-config-volume\") pod \"payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh\" (UID: \"82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8\") " pod="openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh" Oct 30 13:09:27.811086 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:27.808458 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v8cj\" (UniqueName: \"kubernetes.io/projected/82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8-kube-api-access-6v8cj\") pod \"payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh\" (UID: \"82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8\") " pod="openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh" Oct 30 13:09:27.912734 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:27.912695 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v8cj\" (UniqueName: \"kubernetes.io/projected/82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8-kube-api-access-6v8cj\") pod \"payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh\" (UID: \"82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8\") " pod="openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh" Oct 30 13:09:27.925359 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:27.925275 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8-plugins-config-volume\") pod \"payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh\" (UID: \"82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8\") " pod="openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh" Oct 30 13:09:27.927568 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:27.927535 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8-plugins-config-volume\") pod \"payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh\" (UID: \"82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8\") " pod="openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh" Oct 30 13:09:27.938376 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:27.937711 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v8cj\" (UniqueName: \"kubernetes.io/projected/82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8-kube-api-access-6v8cj\") pod \"payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh\" (UID: \"82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8\") " pod="openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh" Oct 30 13:09:28.167714 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:28.167666 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh" Oct 30 13:09:28.483186 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:28.483146 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh"] Oct 30 13:09:28.484427 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:09:28.484399 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f6e537_8a1b_42bc_9462_a3b5d5cbdcf8.slice/crio-78f4e33393f175b21e754e650c65984c75138c8506cd4511cacfddb9c8c5039c WatchSource:0}: Error finding container 78f4e33393f175b21e754e650c65984c75138c8506cd4511cacfddb9c8c5039c: Status 404 returned error can't find the container with id 78f4e33393f175b21e754e650c65984c75138c8506cd4511cacfddb9c8c5039c Oct 30 13:09:29.073422 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:29.071889 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh" event={"ID":"82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8","Type":"ContainerStarted","Data":"676a786b6d485877477c2e0d93e1a0ab196944dc4048470140d6d0dc3738e77f"} Oct 30 13:09:29.073422 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:29.072028 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh" event={"ID":"82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8","Type":"ContainerStarted","Data":"78f4e33393f175b21e754e650c65984c75138c8506cd4511cacfddb9c8c5039c"} Oct 30 13:09:29.073422 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:29.073300 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh" Oct 30 13:09:29.122733 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:29.122673 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh" podStartSLOduration=2.122658631 podStartE2EDuration="2.122658631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:29.118828395 +0000 UTC m=+1062.942056726" watchObservedRunningTime="2026-10-30 13:09:29.122658631 +0000 UTC m=+1062.945886940" Oct 30 13:09:29.280880 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:29.280851 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8648-e2d0d0" Oct 30 13:09:29.469691 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:29.469661 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/976ca170-0822-440c-9e3e-e4598f72185a-kube-api-access-95zds\" (UniqueName: \"kubernetes.io/projected/976ca170-0822-440c-9e3e-e4598f72185a-kube-api-access-95zds\") pod \"976ca170-0822-440c-9e3e-e4598f72185a\" (UID: \"976ca170-0822-440c-9e3e-e4598f72185a\") " Oct 30 13:09:29.471983 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:29.471922 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976ca170-0822-440c-9e3e-e4598f72185a-kube-api-access-95zds" pod "976ca170-0822-440c-9e3e-e4598f72185a" (UID: "976ca170-0822-440c-9e3e-e4598f72185a"). InnerVolumeSpecName "kube-api-access-95zds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:29.572409 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:29.572321 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95zds\" (UniqueName: \"kubernetes.io/projected/976ca170-0822-440c-9e3e-e4598f72185a-kube-api-access-95zds\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:09:29.703955 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:09:29.703907 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:09:30.298974 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:30.295742 4503 generic.go:356] "Generic (PLEG): container finished" podID="976ca170-0822-440c-9e3e-e4598f72185a" containerID="feb071fef1e162fee004d1d65616db19eed7ee394ec91c093eb695ec2df7344f" exitCode=137 Oct 30 13:09:30.298974 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:30.296870 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8648-e2d0d0" Oct 30 13:09:30.301284 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:30.300052 4503 scope.go:122] "RemoveContainer" containerID="feb071fef1e162fee004d1d65616db19eed7ee394ec91c093eb695ec2df7344f" Oct 30 13:09:30.486587 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:09:30.486547 4503 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8648-e2d0d0\" is forbidden: User \"system:node:ip-10-0-130-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-200.ec2.internal' and this object" podUID="976ca170-0822-440c-9e3e-e4598f72185a" pod="opendatahub/test-curl-8648-e2d0d0" Oct 30 13:09:30.700280 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:30.700236 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="976ca170-0822-440c-9e3e-e4598f72185a" path="/var/lib/kubelet/pods/976ca170-0822-440c-9e3e-e4598f72185a/volumes" Oct 30 13:09:30.720381 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:30.718144 4503 scope.go:122] "RemoveContainer" containerID="feb071fef1e162fee004d1d65616db19eed7ee394ec91c093eb695ec2df7344f" Oct 30 13:09:30.731117 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:09:30.730108 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb071fef1e162fee004d1d65616db19eed7ee394ec91c093eb695ec2df7344f\": container with ID starting with feb071fef1e162fee004d1d65616db19eed7ee394ec91c093eb695ec2df7344f not found: ID does not exist" containerID="feb071fef1e162fee004d1d65616db19eed7ee394ec91c093eb695ec2df7344f" Oct 30 13:09:30.731117 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:30.730164 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb071fef1e162fee004d1d65616db19eed7ee394ec91c093eb695ec2df7344f"} err="failed to get container status \"feb071fef1e162fee004d1d65616db19eed7ee394ec91c093eb695ec2df7344f\": rpc error: code = NotFound desc = could not find container \"feb071fef1e162fee004d1d65616db19eed7ee394ec91c093eb695ec2df7344f\": container with ID starting with feb071fef1e162fee004d1d65616db19eed7ee394ec91c093eb695ec2df7344f not found: ID does not exist" Oct 30 13:09:31.428962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:31.425251 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:32.115164 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:32.115103 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:32.116132 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:32.116091 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:32.425574 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:32.425532 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:32.427661 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:32.426326 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:09:33.464971 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:33.464915 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:09:33.465624 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:33.465588 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" podUID="72c4c258-7da6-40a5-82f5-618f8bea345f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:33.537099 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:33.537041 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:33.566448 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:33.560508 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 13:09:33.602971 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:33.589306 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 13:09:33.709143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:33.642399 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:09:33.709143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:33.642500 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:09:34.599271 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:34.599229 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 13:09:34.633967 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:34.604056 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:34.650237 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:34.632774 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:35.620964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:35.618100 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:35.814071 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:35.734774 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 13:09:35.845057 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:35.747527 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:36.333336 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:36.333290 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:09:36.456964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:36.453118 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:09:36.727959 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:36.727458 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:09:36.818101 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:36.818066 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:09:36.818394 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:36.818353 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" podUID="c1d0fc5e-a51f-490d-892a-809a3464c0b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:40.603809 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.603767 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2"] Oct 30 13:09:40.610527 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.610502 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.633782 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.633752 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2"] Oct 30 13:09:40.641011 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.640885 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-253f7c37-openshift-default-dockercfg-7rl9t\"" Oct 30 13:09:40.683844 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.683800 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7"] Oct 30 13:09:40.688601 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.688576 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" Oct 30 13:09:40.706617 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.706578 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7"] Oct 30 13:09:40.727124 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.727093 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-751d84fb-serving-cert\"" Oct 30 13:09:40.784336 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.784296 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-credential-socket\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.784544 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.784347 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-workload-socket\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.784544 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.784488 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-workload-certs\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.784544 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.784506 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p96qc\" (UniqueName: \"kubernetes.io/projected/0549ac29-5939-4f6c-aced-091b158f4afd-kube-api-access-p96qc\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.784684 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.784618 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-istio-envoy\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.784800 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.784781 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0549ac29-5939-4f6c-aced-091b158f4afd-istio-token\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.784904 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.784888 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0549ac29-5939-4f6c-aced-091b158f4afd-istio-podinfo\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.784972 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.784921 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-istio-data\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.784972 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.784962 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0549ac29-5939-4f6c-aced-091b158f4afd-istiod-ca-cert\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.887537 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.887434 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-workload-certs\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.926101 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.926063 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p96qc\" (UniqueName: \"kubernetes.io/projected/0549ac29-5939-4f6c-aced-091b158f4afd-kube-api-access-p96qc\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.926401 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.926370 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-istio-envoy\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.926656 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.926633 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-openshift-service-ca\") pod \"maas-api-e2e-ait-751d84fb-677d468bb5-479r7\" (UID: \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" Oct 30 13:09:40.926957 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.926923 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-maas-api-tls\") pod \"maas-api-e2e-ait-751d84fb-677d468bb5-479r7\" (UID: \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" Oct 30 13:09:40.928129 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.928106 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0549ac29-5939-4f6c-aced-091b158f4afd-istio-token\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.928498 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.928466 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0549ac29-5939-4f6c-aced-091b158f4afd-istio-podinfo\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.928647 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.928629 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhjt6\" (UniqueName: \"kubernetes.io/projected/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-kube-api-access-lhjt6\") pod \"maas-api-e2e-ait-751d84fb-677d468bb5-479r7\" (UID: \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" Oct 30 13:09:40.928786 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.928767 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-istio-data\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.929021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.928997 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0549ac29-5939-4f6c-aced-091b158f4afd-istiod-ca-cert\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.929351 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.929319 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-credential-socket\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.929517 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.929499 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-metrics-certs\") pod \"maas-api-e2e-ait-751d84fb-677d468bb5-479r7\" (UID: \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" Oct 30 13:09:40.929646 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.929629 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-workload-socket\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.930360 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.930334 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-workload-socket\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.950053 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.888374 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-workload-certs\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.950280 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.948566 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-credential-socket\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.950400 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.948855 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-istio-data\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.950528 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.949982 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0549ac29-5939-4f6c-aced-091b158f4afd-istiod-ca-cert\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.972349 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.971800 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-istio-envoy\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.973351 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.972864 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p96qc\" (UniqueName: \"kubernetes.io/projected/0549ac29-5939-4f6c-aced-091b158f4afd-kube-api-access-p96qc\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.980584 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.980507 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0549ac29-5939-4f6c-aced-091b158f4afd-istio-podinfo\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:40.998764 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:40.998470 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0549ac29-5939-4f6c-aced-091b158f4afd-istio-token\") pod \"e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:41.045522 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:41.030886 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhjt6\" (UniqueName: \"kubernetes.io/projected/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-kube-api-access-lhjt6\") pod \"maas-api-e2e-ait-751d84fb-677d468bb5-479r7\" (UID: \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" Oct 30 13:09:41.045522 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:41.031068 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-metrics-certs\") pod \"maas-api-e2e-ait-751d84fb-677d468bb5-479r7\" (UID: \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" Oct 30 13:09:41.045522 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:41.031233 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-openshift-service-ca\") pod \"maas-api-e2e-ait-751d84fb-677d468bb5-479r7\" (UID: \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" Oct 30 13:09:41.045522 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:41.031272 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-maas-api-tls\") pod \"maas-api-e2e-ait-751d84fb-677d468bb5-479r7\" (UID: \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" Oct 30 13:09:41.045522 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:41.034435 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-openshift-service-ca\") pod \"maas-api-e2e-ait-751d84fb-677d468bb5-479r7\" (UID: \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" Oct 30 13:09:41.062389 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:41.054594 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-metrics-certs\") pod \"maas-api-e2e-ait-751d84fb-677d468bb5-479r7\" (UID: \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" Oct 30 13:09:41.086596 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:41.086527 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-maas-api-tls\") pod \"maas-api-e2e-ait-751d84fb-677d468bb5-479r7\" (UID: \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" Oct 30 13:09:41.113070 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:41.105594 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhjt6\" (UniqueName: \"kubernetes.io/projected/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-kube-api-access-lhjt6\") pod \"maas-api-e2e-ait-751d84fb-677d468bb5-479r7\" (UID: \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" Oct 30 13:09:41.247867 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:41.247832 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:41.317202 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:41.317172 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" Oct 30 13:09:41.522473 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:41.522391 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh" Oct 30 13:09:41.654713 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:41.654678 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2"] Oct 30 13:09:41.656435 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:09:41.656411 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0549ac29_5939_4f6c_aced_091b158f4afd.slice/crio-b3bccbea15b4857e762d15f16b00fbb9336ecad05514f3b0dd85e1dd3fd8a583 WatchSource:0}: Error finding container b3bccbea15b4857e762d15f16b00fbb9336ecad05514f3b0dd85e1dd3fd8a583: Status 404 returned error can't find the container with id b3bccbea15b4857e762d15f16b00fbb9336ecad05514f3b0dd85e1dd3fd8a583 Oct 30 13:09:41.662872 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:41.662832 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:41.663079 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:41.662966 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:41.663150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:41.663088 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:41.665766 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:41.665687 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7"] Oct 30 13:09:41.668868 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:09:41.668843 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a867e6d_42d0_4e78_bbd6_e4521a8e9b92.slice/crio-cca792fc99fbfc8910bd748e63c5d3c3a5e219ff53a1ee5f8266e27a6e1dd184 WatchSource:0}: Error finding container cca792fc99fbfc8910bd748e63c5d3c3a5e219ff53a1ee5f8266e27a6e1dd184: Status 404 returned error can't find the container with id cca792fc99fbfc8910bd748e63c5d3c3a5e219ff53a1ee5f8266e27a6e1dd184 Oct 30 13:09:42.454260 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:42.454209 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" event={"ID":"0549ac29-5939-4f6c-aced-091b158f4afd","Type":"ContainerStarted","Data":"bdd197d00bc095e8f4d4a0a9a86e5c64e90b853d648b65df3aea523182e18ea8"} Oct 30 13:09:42.454521 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:42.454502 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" event={"ID":"0549ac29-5939-4f6c-aced-091b158f4afd","Type":"ContainerStarted","Data":"b3bccbea15b4857e762d15f16b00fbb9336ecad05514f3b0dd85e1dd3fd8a583"} Oct 30 13:09:42.456603 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:42.456578 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" event={"ID":"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92","Type":"ContainerStarted","Data":"734b1342f573163bd3b9b2272ce23bfd4b167dab866d3b422969fd8d63f75885"} Oct 30 13:09:42.456792 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:42.456767 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" event={"ID":"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92","Type":"ContainerStarted","Data":"cca792fc99fbfc8910bd748e63c5d3c3a5e219ff53a1ee5f8266e27a6e1dd184"} Oct 30 13:09:42.456907 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:42.456806 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" Oct 30 13:09:42.502885 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:42.502833 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" podStartSLOduration=2.502817519 podStartE2EDuration="2.502817519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:42.502065241 +0000 UTC m=+1076.325293517" watchObservedRunningTime="2026-10-30 13:09:42.502817519 +0000 UTC m=+1076.326045799" Oct 30 13:09:42.553324 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:42.553251 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" podStartSLOduration=2.19248989 podStartE2EDuration="2.553233584s" totalImagesPullingTime="360.743694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:42.54721827 +0000 UTC m=+1076.370446556" watchObservedRunningTime="2026-10-30 13:09:42.553233584 +0000 UTC m=+1076.376461862" Oct 30 13:09:43.250545 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:43.250507 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:43.257121 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:43.257089 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:43.476843 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:43.476808 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:43.479123 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:43.479092 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:09:46.076804 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:46.076771 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b"] Oct 30 13:09:46.084627 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:46.084593 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b" Oct 30 13:09:46.103473 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:46.103442 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b"] Oct 30 13:09:46.108450 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:46.108423 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-253f7c37\"" Oct 30 13:09:46.108609 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:46.108463 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-253f7c37-dockercfg-hh2hz\"" Oct 30 13:09:46.214293 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:46.207263 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr6pb\" (UniqueName: \"kubernetes.io/projected/08339a18-3dd6-4f65-bff1-3c3b83d8ea5b-kube-api-access-fr6pb\") pod \"payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b\" (UID: \"08339a18-3dd6-4f65-bff1-3c3b83d8ea5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b" Oct 30 13:09:46.214750 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:46.214712 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08339a18-3dd6-4f65-bff1-3c3b83d8ea5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b\" (UID: \"08339a18-3dd6-4f65-bff1-3c3b83d8ea5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b" Oct 30 13:09:46.559560 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:46.559515 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr6pb\" (UniqueName: \"kubernetes.io/projected/08339a18-3dd6-4f65-bff1-3c3b83d8ea5b-kube-api-access-fr6pb\") pod \"payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b\" (UID: \"08339a18-3dd6-4f65-bff1-3c3b83d8ea5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b" Oct 30 13:09:46.559771 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:46.559591 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08339a18-3dd6-4f65-bff1-3c3b83d8ea5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b\" (UID: \"08339a18-3dd6-4f65-bff1-3c3b83d8ea5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b" Oct 30 13:09:46.604046 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:46.600409 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-253f7c37\"" Oct 30 13:09:46.608449 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:46.608404 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08339a18-3dd6-4f65-bff1-3c3b83d8ea5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b\" (UID: \"08339a18-3dd6-4f65-bff1-3c3b83d8ea5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b" Oct 30 13:09:46.613060 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:46.613034 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr6pb\" (UniqueName: \"kubernetes.io/projected/08339a18-3dd6-4f65-bff1-3c3b83d8ea5b-kube-api-access-fr6pb\") pod \"payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b\" (UID: \"08339a18-3dd6-4f65-bff1-3c3b83d8ea5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b" Oct 30 13:09:46.715415 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:46.715384 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-253f7c37-dockercfg-hh2hz\"" Oct 30 13:09:46.722620 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:46.722593 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b" Oct 30 13:09:47.134988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:47.134909 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b"] Oct 30 13:09:47.135421 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:09:47.135273 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08339a18_3dd6_4f65_bff1_3c3b83d8ea5b.slice/crio-439f3d27451fa49593367a553d716caee98917bf1a54e78e9825f5ac45a8da71 WatchSource:0}: Error finding container 439f3d27451fa49593367a553d716caee98917bf1a54e78e9825f5ac45a8da71: Status 404 returned error can't find the container with id 439f3d27451fa49593367a553d716caee98917bf1a54e78e9825f5ac45a8da71 Oct 30 13:09:47.624016 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:47.623981 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b" event={"ID":"08339a18-3dd6-4f65-bff1-3c3b83d8ea5b","Type":"ContainerStarted","Data":"acde95ca3287bb0a15e1a6ee98df3a71a124293e5d74bda8e52d243da35a0fe7"} Oct 30 13:09:47.624260 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:47.624038 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b" event={"ID":"08339a18-3dd6-4f65-bff1-3c3b83d8ea5b","Type":"ContainerStarted","Data":"439f3d27451fa49593367a553d716caee98917bf1a54e78e9825f5ac45a8da71"} Oct 30 13:09:47.624260 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:47.624064 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b" Oct 30 13:09:47.674174 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:47.674123 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b" podStartSLOduration=1.674107776 podStartE2EDuration="1.674107776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:47.670453841 +0000 UTC m=+1081.493682120" watchObservedRunningTime="2026-10-30 13:09:47.674107776 +0000 UTC m=+1081.497336053" Oct 30 13:09:48.481407 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:48.481373 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" Oct 30 13:09:48.654464 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:48.654430 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv"] Oct 30 13:09:48.680716 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:48.680681 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" Oct 30 13:09:48.681598 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:48.681573 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv"] Oct 30 13:09:48.697265 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:48.697236 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-4a59cbc1-serving-cert\"" Oct 30 13:09:48.728177 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:48.728131 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a3e261-7af6-44ae-8449-bbc8d016f239-maas-api-tls\") pod \"maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv\" (UID: \"04a3e261-7af6-44ae-8449-bbc8d016f239\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" Oct 30 13:09:48.728359 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:48.728210 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwmwg\" (UniqueName: \"kubernetes.io/projected/04a3e261-7af6-44ae-8449-bbc8d016f239-kube-api-access-dwmwg\") pod \"maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv\" (UID: \"04a3e261-7af6-44ae-8449-bbc8d016f239\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" Oct 30 13:09:48.728359 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:48.728342 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a3e261-7af6-44ae-8449-bbc8d016f239-openshift-service-ca\") pod \"maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv\" (UID: \"04a3e261-7af6-44ae-8449-bbc8d016f239\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" Oct 30 13:09:48.728580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:48.728545 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04a3e261-7af6-44ae-8449-bbc8d016f239-metrics-certs\") pod \"maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv\" (UID: \"04a3e261-7af6-44ae-8449-bbc8d016f239\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" Oct 30 13:09:48.829522 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:48.829436 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a3e261-7af6-44ae-8449-bbc8d016f239-maas-api-tls\") pod \"maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv\" (UID: \"04a3e261-7af6-44ae-8449-bbc8d016f239\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" Oct 30 13:09:48.829522 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:48.829496 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwmwg\" (UniqueName: \"kubernetes.io/projected/04a3e261-7af6-44ae-8449-bbc8d016f239-kube-api-access-dwmwg\") pod \"maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv\" (UID: \"04a3e261-7af6-44ae-8449-bbc8d016f239\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" Oct 30 13:09:48.829742 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:48.829622 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a3e261-7af6-44ae-8449-bbc8d016f239-openshift-service-ca\") pod \"maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv\" (UID: \"04a3e261-7af6-44ae-8449-bbc8d016f239\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" Oct 30 13:09:48.829803 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:48.829791 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04a3e261-7af6-44ae-8449-bbc8d016f239-metrics-certs\") pod \"maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv\" (UID: \"04a3e261-7af6-44ae-8449-bbc8d016f239\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" Oct 30 13:09:48.830347 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:48.830321 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a3e261-7af6-44ae-8449-bbc8d016f239-openshift-service-ca\") pod \"maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv\" (UID: \"04a3e261-7af6-44ae-8449-bbc8d016f239\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" Oct 30 13:09:48.831992 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:48.831961 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a3e261-7af6-44ae-8449-bbc8d016f239-maas-api-tls\") pod \"maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv\" (UID: \"04a3e261-7af6-44ae-8449-bbc8d016f239\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" Oct 30 13:09:48.832160 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:48.832143 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04a3e261-7af6-44ae-8449-bbc8d016f239-metrics-certs\") pod \"maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv\" (UID: \"04a3e261-7af6-44ae-8449-bbc8d016f239\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" Oct 30 13:09:48.856716 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:48.856684 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwmwg\" (UniqueName: \"kubernetes.io/projected/04a3e261-7af6-44ae-8449-bbc8d016f239-kube-api-access-dwmwg\") pod \"maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv\" (UID: \"04a3e261-7af6-44ae-8449-bbc8d016f239\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" Oct 30 13:09:49.015691 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:49.015648 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" Oct 30 13:09:49.544970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:49.542182 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv"] Oct 30 13:09:49.675192 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:49.674463 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" event={"ID":"04a3e261-7af6-44ae-8449-bbc8d016f239","Type":"ContainerStarted","Data":"07290b5f7c686a25bce4bcfb3ba441db5fc5e0d262e71119915677457b059da2"} Oct 30 13:09:50.721963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.719793 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" event={"ID":"04a3e261-7af6-44ae-8449-bbc8d016f239","Type":"ContainerStarted","Data":"972201b292958fa2e9d5dcdd0f63513375c1e1e2b3295432d52d766005f068e5"} Oct 30 13:09:50.721963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.720011 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" Oct 30 13:09:50.731408 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.723421 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp"] Oct 30 13:09:50.737984 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.731886 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.764765 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.749529 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-envoy\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.764765 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.749588 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-workload-certs\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.764765 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.749627 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-workload-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.764765 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.749758 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rngnt\" (UniqueName: \"kubernetes.io/projected/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-kube-api-access-rngnt\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.764765 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.749815 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-credential-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.764765 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.749996 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-token\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.764765 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.750038 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-data\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.764765 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.750068 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istiod-ca-cert\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.764765 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.750121 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-podinfo\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.841184 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.839356 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp"] Oct 30 13:09:50.863973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.856074 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" podStartSLOduration=2.365306434 podStartE2EDuration="2.856053133s" totalImagesPullingTime="490.746699ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:50.79151699 +0000 UTC m=+1084.614745267" watchObservedRunningTime="2026-10-30 13:09:50.856053133 +0000 UTC m=+1084.679281413" Oct 30 13:09:50.903963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.903381 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-envoy\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.903963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.903445 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-workload-certs\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.903963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.903485 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-workload-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.903963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.903601 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rngnt\" (UniqueName: \"kubernetes.io/projected/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-kube-api-access-rngnt\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.903963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.903658 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-credential-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.903963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.903779 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-token\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.903963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.903818 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-data\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.903963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.903849 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istiod-ca-cert\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.903963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.903894 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-podinfo\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.932401 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.904526 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-workload-certs\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.932401 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.913637 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-credential-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.932401 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.913929 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-data\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.932401 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.914606 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istiod-ca-cert\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.932401 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.914862 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-workload-socket\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:50.960966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:50.954280 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-envoy\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:51.066010 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:51.063298 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-podinfo\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:51.084001 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:51.066413 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rngnt\" (UniqueName: \"kubernetes.io/projected/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-kube-api-access-rngnt\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:51.097033 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:51.096641 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-token\") pod \"e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:51.257249 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:51.257203 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:51.850960 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:51.839109 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:51.850960 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:51.839210 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:51.850960 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:51.839253 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:51.855154 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:51.854479 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp"] Oct 30 13:09:51.902582 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:51.902528 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" event={"ID":"81a74b8e-b1f2-467b-bb2c-6b0029e464f4","Type":"ContainerStarted","Data":"4685e9707abc716fc8f2444eda968f643152cf71764a3cf19b4d672288843a40"} Oct 30 13:09:52.911445 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:52.911403 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" event={"ID":"81a74b8e-b1f2-467b-bb2c-6b0029e464f4","Type":"ContainerStarted","Data":"4af7d550e92bda682187fd81e5821e501d48dec4dd0f744701ee746d1e403f05"} Oct 30 13:09:52.980964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:52.980266 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" podStartSLOduration=2.980247517 podStartE2EDuration="2.980247517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:52.976444289 +0000 UTC m=+1086.799672599" watchObservedRunningTime="2026-10-30 13:09:52.980247517 +0000 UTC m=+1086.803475797" Oct 30 13:09:53.267408 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:53.267312 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:53.271080 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:53.271048 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:09:53.271306 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:53.271282 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" podUID="81a74b8e-b1f2-467b-bb2c-6b0029e464f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 13:09:54.268967 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:54.268850 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:09:54.268967 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:54.268914 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" podUID="81a74b8e-b1f2-467b-bb2c-6b0029e464f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 13:09:55.263964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:55.258284 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:09:55.263964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:55.258347 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" podUID="81a74b8e-b1f2-467b-bb2c-6b0029e464f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 13:09:56.266963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:56.263045 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:09:56.266963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:56.263127 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" podUID="81a74b8e-b1f2-467b-bb2c-6b0029e464f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 13:09:56.908059 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:56.908028 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" Oct 30 13:09:57.268023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:57.265617 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:09:57.268023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:57.265677 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" podUID="81a74b8e-b1f2-467b-bb2c-6b0029e464f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 13:09:58.258502 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:58.258459 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 13:09:58.258696 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:58.258531 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" podUID="81a74b8e-b1f2-467b-bb2c-6b0029e464f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 13:09:58.635006 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:58.634921 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b" Oct 30 13:09:59.473028 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:59.470885 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:59.473028 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:59.472650 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:59.476657 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:09:59.476247 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:09:59.720083 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:09:59.720044 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:10:03.395384 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:03.395345 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z"] Oct 30 13:10:03.401390 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:03.401364 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" Oct 30 13:10:03.425003 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:03.420825 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-253f7c37-serving-cert\"" Oct 30 13:10:03.425003 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:03.423644 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z"] Oct 30 13:10:03.545783 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:03.545748 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd6e615-1e54-4058-b289-a1fa783e00a7-openshift-service-ca\") pod \"maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z\" (UID: \"acd6e615-1e54-4058-b289-a1fa783e00a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" Oct 30 13:10:03.545994 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:03.545963 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4h7n\" (UniqueName: \"kubernetes.io/projected/acd6e615-1e54-4058-b289-a1fa783e00a7-kube-api-access-m4h7n\") pod \"maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z\" (UID: \"acd6e615-1e54-4058-b289-a1fa783e00a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" Oct 30 13:10:03.546063 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:03.545997 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acd6e615-1e54-4058-b289-a1fa783e00a7-metrics-certs\") pod \"maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z\" (UID: \"acd6e615-1e54-4058-b289-a1fa783e00a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" Oct 30 13:10:03.546176 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:03.546154 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acd6e615-1e54-4058-b289-a1fa783e00a7-maas-api-tls\") pod \"maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z\" (UID: \"acd6e615-1e54-4058-b289-a1fa783e00a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" Oct 30 13:10:03.648282 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:03.648184 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4h7n\" (UniqueName: \"kubernetes.io/projected/acd6e615-1e54-4058-b289-a1fa783e00a7-kube-api-access-m4h7n\") pod \"maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z\" (UID: \"acd6e615-1e54-4058-b289-a1fa783e00a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" Oct 30 13:10:03.648282 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:03.648256 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acd6e615-1e54-4058-b289-a1fa783e00a7-metrics-certs\") pod \"maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z\" (UID: \"acd6e615-1e54-4058-b289-a1fa783e00a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" Oct 30 13:10:03.648539 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:03.648361 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acd6e615-1e54-4058-b289-a1fa783e00a7-maas-api-tls\") pod \"maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z\" (UID: \"acd6e615-1e54-4058-b289-a1fa783e00a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" Oct 30 13:10:03.648539 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:03.648505 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd6e615-1e54-4058-b289-a1fa783e00a7-openshift-service-ca\") pod \"maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z\" (UID: \"acd6e615-1e54-4058-b289-a1fa783e00a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" Oct 30 13:10:03.649278 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:03.649250 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd6e615-1e54-4058-b289-a1fa783e00a7-openshift-service-ca\") pod \"maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z\" (UID: \"acd6e615-1e54-4058-b289-a1fa783e00a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" Oct 30 13:10:03.651372 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:03.651350 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acd6e615-1e54-4058-b289-a1fa783e00a7-metrics-certs\") pod \"maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z\" (UID: \"acd6e615-1e54-4058-b289-a1fa783e00a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" Oct 30 13:10:03.651504 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:03.651469 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acd6e615-1e54-4058-b289-a1fa783e00a7-maas-api-tls\") pod \"maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z\" (UID: \"acd6e615-1e54-4058-b289-a1fa783e00a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" Oct 30 13:10:03.704959 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:03.700666 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4h7n\" (UniqueName: \"kubernetes.io/projected/acd6e615-1e54-4058-b289-a1fa783e00a7-kube-api-access-m4h7n\") pod \"maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z\" (UID: \"acd6e615-1e54-4058-b289-a1fa783e00a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" Oct 30 13:10:03.728600 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:03.728304 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" Oct 30 13:10:04.195207 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:10:04.195177 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd6e615_1e54_4058_b289_a1fa783e00a7.slice/crio-1f9b66d8965a01ca33f91d38e5090980e064042dfb752c9de6cebf28eb42e5ad WatchSource:0}: Error finding container 1f9b66d8965a01ca33f91d38e5090980e064042dfb752c9de6cebf28eb42e5ad: Status 404 returned error can't find the container with id 1f9b66d8965a01ca33f91d38e5090980e064042dfb752c9de6cebf28eb42e5ad Oct 30 13:10:04.201216 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:04.201188 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z"] Oct 30 13:10:05.084967 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:05.076070 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" event={"ID":"acd6e615-1e54-4058-b289-a1fa783e00a7","Type":"ContainerStarted","Data":"f3e1619648b759e9ba0cd014ca13f611108a35d159beca27270c41eda764798f"} Oct 30 13:10:05.084967 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:05.076177 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" event={"ID":"acd6e615-1e54-4058-b289-a1fa783e00a7","Type":"ContainerStarted","Data":"1f9b66d8965a01ca33f91d38e5090980e064042dfb752c9de6cebf28eb42e5ad"} Oct 30 13:10:05.084967 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:05.077351 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" Oct 30 13:10:05.137964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:05.137323 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" podStartSLOduration=1.779926932 podStartE2EDuration="2.13729846s" totalImagesPullingTime="357.371528ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:05.124373691 +0000 UTC m=+1098.947601971" watchObservedRunningTime="2026-10-30 13:10:05.13729846 +0000 UTC m=+1098.960526744" Oct 30 13:10:09.044423 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:09.044385 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7"] Oct 30 13:10:09.056497 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:09.056466 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cb89605-c5gt7"] Oct 30 13:10:10.377391 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:10.377341 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71e7cebb-eea5-44c2-a772-89c46a9b63e2" path="/var/lib/kubelet/pods/71e7cebb-eea5-44c2-a772-89c46a9b63e2/volumes" Oct 30 13:10:12.109974 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:12.108444 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" Oct 30 13:10:15.554422 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:15.554369 4503 patch_prober.go:29] interesting pod/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:10:15.555059 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:15.555030 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" podUID="81a74b8e-b1f2-467b-bb2c-6b0029e464f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:15.628996 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:15.628348 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:15.738975 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:15.687213 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:20.602676 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.602630 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj"] Oct 30 13:10:20.609989 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.609961 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.668285 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.668198 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj"] Oct 30 13:10:20.674895 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.674855 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.675109 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.675073 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.675174 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.675120 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.675174 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.675149 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzkxn\" (UniqueName: \"kubernetes.io/projected/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-kube-api-access-hzkxn\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.675277 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.675231 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.675277 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.675249 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.675379 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.675293 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.675379 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.675332 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.675482 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.675386 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.777068 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.777022 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.777068 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.777075 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.777331 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.777142 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.777331 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.777206 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.777331 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.777290 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.777484 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.777362 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.777544 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.777516 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.777604 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.777558 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.777604 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.777580 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzkxn\" (UniqueName: \"kubernetes.io/projected/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-kube-api-access-hzkxn\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.777712 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.777650 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.777864 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.777839 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.778137 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.778102 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.778137 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.778114 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.779155 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.779028 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.780428 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.780406 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.780827 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.780806 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.803761 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.803731 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.803973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.803929 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzkxn\" (UniqueName: \"kubernetes.io/projected/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-kube-api-access-hzkxn\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:20.930899 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:20.930862 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:21.276693 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:10:21.276659 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a2a7df_fe4f_4055_9de6_b1acaae0db54.slice/crio-798d75960333dd75aecf60e2924b2c32c823772a8c31a57399f23725cd9c1b86 WatchSource:0}: Error finding container 798d75960333dd75aecf60e2924b2c32c823772a8c31a57399f23725cd9c1b86: Status 404 returned error can't find the container with id 798d75960333dd75aecf60e2924b2c32c823772a8c31a57399f23725cd9c1b86 Oct 30 13:10:21.277169 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:21.277142 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj"] Oct 30 13:10:21.280710 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:21.280662 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:21.280849 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:21.280747 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:21.280849 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:21.280800 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:21.786303 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:21.776979 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" event={"ID":"f6a2a7df-fe4f-4055-9de6-b1acaae0db54","Type":"ContainerStarted","Data":"593a2b8b4297111030ace0025a29f43d9f5ccb30aa1e5ee508db2dd9fcc24cd4"} Oct 30 13:10:21.786303 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:21.777083 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" event={"ID":"f6a2a7df-fe4f-4055-9de6-b1acaae0db54","Type":"ContainerStarted","Data":"798d75960333dd75aecf60e2924b2c32c823772a8c31a57399f23725cd9c1b86"} Oct 30 13:10:21.828579 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:21.828522 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" podStartSLOduration=1.828507594 podStartE2EDuration="1.828507594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:21.825515514 +0000 UTC m=+1115.648743803" watchObservedRunningTime="2026-10-30 13:10:21.828507594 +0000 UTC m=+1115.651735872" Oct 30 13:10:21.931849 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:21.931807 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:22.933404 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:22.933374 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 13:10:22.933813 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:22.933426 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" podUID="f6a2a7df-fe4f-4055-9de6-b1acaae0db54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 13:10:23.931966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:23.931919 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 13:10:23.932143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:23.931994 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" podUID="f6a2a7df-fe4f-4055-9de6-b1acaae0db54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 13:10:24.909756 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:24.909718 4503 generic.go:356] "Generic (PLEG): container finished" podID="1cc2d750-6f71-4f16-8228-82e6390d9bcf" containerID="a0bb50ffd774b138f89e420793c04af18f1c7fb10cd349438c0ced1c6eeb1c25" exitCode=0 Oct 30 13:10:24.910229 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:24.909790 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" event={"ID":"1cc2d750-6f71-4f16-8228-82e6390d9bcf","Type":"ContainerDied","Data":"a0bb50ffd774b138f89e420793c04af18f1c7fb10cd349438c0ced1c6eeb1c25"} Oct 30 13:10:24.910340 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:24.910325 4503 scope.go:122] "RemoveContainer" containerID="a0bb50ffd774b138f89e420793c04af18f1c7fb10cd349438c0ced1c6eeb1c25" Oct 30 13:10:24.912065 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:24.912036 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:24.912184 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:24.912115 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:24.912184 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:24.912150 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:24.931700 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:24.931676 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 13:10:24.931790 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:24.931722 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" podUID="f6a2a7df-fe4f-4055-9de6-b1acaae0db54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 13:10:25.933972 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:25.933481 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 13:10:25.933972 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:25.933545 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" podUID="f6a2a7df-fe4f-4055-9de6-b1acaae0db54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 13:10:25.951959 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:25.936222 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" event={"ID":"1cc2d750-6f71-4f16-8228-82e6390d9bcf","Type":"ContainerStarted","Data":"7f3d4ae25203dd801808fbc38b71461639397b9ed2b3029e17623a36fcbb549b"} Oct 30 13:10:26.827846 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:26.827793 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:10:26.828080 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:26.827912 4503 patch_prober.go:29] interesting pod/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:10:26.828080 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:26.827975 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" podUID="1cc2d750-6f71-4f16-8228-82e6390d9bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 13:10:26.887996 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:26.887179 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh"] Oct 30 13:10:26.931248 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:26.931211 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 13:10:26.931421 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:26.931292 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" podUID="f6a2a7df-fe4f-4055-9de6-b1acaae0db54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 13:10:27.856915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:27.856868 4503 patch_prober.go:29] interesting pod/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:10:27.857506 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:27.857474 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" podUID="1cc2d750-6f71-4f16-8228-82e6390d9bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 13:10:27.882268 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:27.882230 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7"] Oct 30 13:10:27.882858 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:27.882795 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" podUID="7a867e6d-42d0-4e78-bbd6-e4521a8e9b92" containerName="maas-api" containerID="cri-o://734b1342f573163bd3b9b2272ce23bfd4b167dab866d3b422969fd8d63f75885" gracePeriod=30 Oct 30 13:10:28.109376 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:28.079814 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:28.118806 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:28.088150 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 13:10:28.119046 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:28.119019 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" podUID="f6a2a7df-fe4f-4055-9de6-b1acaae0db54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 13:10:28.510082 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:28.508420 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" podUID="7a867e6d-42d0-4e78-bbd6-e4521a8e9b92" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.85:8443/health\": dial tcp 10.133.0.85:8443: connect: connection refused" Oct 30 13:10:28.843466 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:28.840030 4503 patch_prober.go:29] interesting pod/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 13:10:28.843466 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:28.840101 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" podUID="1cc2d750-6f71-4f16-8228-82e6390d9bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 13:10:28.933039 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:28.932622 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 13:10:28.933039 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:28.932683 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" podUID="f6a2a7df-fe4f-4055-9de6-b1acaae0db54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 13:10:29.220966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:29.219235 4503 generic.go:356] "Generic (PLEG): container finished" podID="7a867e6d-42d0-4e78-bbd6-e4521a8e9b92" containerID="734b1342f573163bd3b9b2272ce23bfd4b167dab866d3b422969fd8d63f75885" exitCode=0 Oct 30 13:10:29.220966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:29.219332 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" event={"ID":"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92","Type":"ContainerDied","Data":"734b1342f573163bd3b9b2272ce23bfd4b167dab866d3b422969fd8d63f75885"} Oct 30 13:10:29.229933 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:29.229688 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" Oct 30 13:10:29.404964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:29.404497 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-kube-api-access-lhjt6\" (UniqueName: \"kubernetes.io/projected/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-kube-api-access-lhjt6\") pod \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\" (UID: \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\") " Oct 30 13:10:29.404964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:29.404663 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-openshift-service-ca\") pod \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\" (UID: \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\") " Oct 30 13:10:29.404964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:29.404788 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-maas-api-tls\") pod \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\" (UID: \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\") " Oct 30 13:10:29.404964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:29.404923 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-metrics-certs\") pod \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\" (UID: \"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92\") " Oct 30 13:10:29.428011 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:29.423174 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-openshift-service-ca" pod "7a867e6d-42d0-4e78-bbd6-e4521a8e9b92" (UID: "7a867e6d-42d0-4e78-bbd6-e4521a8e9b92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:29.530412 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:29.529733 4503 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-openshift-service-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:29.533124 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:29.530867 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-maas-api-tls" pod "7a867e6d-42d0-4e78-bbd6-e4521a8e9b92" (UID: "7a867e6d-42d0-4e78-bbd6-e4521a8e9b92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:29.534833 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:29.534787 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-metrics-certs" pod "7a867e6d-42d0-4e78-bbd6-e4521a8e9b92" (UID: "7a867e6d-42d0-4e78-bbd6-e4521a8e9b92"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:29.536153 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:29.536113 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-kube-api-access-lhjt6" pod "7a867e6d-42d0-4e78-bbd6-e4521a8e9b92" (UID: "7a867e6d-42d0-4e78-bbd6-e4521a8e9b92"). InnerVolumeSpecName "kube-api-access-lhjt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:29.542307 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:29.542280 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j"] Oct 30 13:10:29.542706 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:29.542677 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" podUID="1cc2d750-6f71-4f16-8228-82e6390d9bcf" containerName="istio-proxy" containerID="cri-o://7f3d4ae25203dd801808fbc38b71461639397b9ed2b3029e17623a36fcbb549b" gracePeriod=30 Oct 30 13:10:29.659084 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:29.656373 4503 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-metrics-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:29.659084 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:29.656415 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhjt6\" (UniqueName: \"kubernetes.io/projected/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-kube-api-access-lhjt6\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:29.659084 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:29.656432 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92-maas-api-tls\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:29.987037 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:29.952767 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 13:10:29.987037 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:29.952831 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" podUID="f6a2a7df-fe4f-4055-9de6-b1acaae0db54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 13:10:30.220288 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:10:30.220241 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:10:30.233636 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:30.233607 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" Oct 30 13:10:30.233813 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:30.233615 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7" event={"ID":"7a867e6d-42d0-4e78-bbd6-e4521a8e9b92","Type":"ContainerDied","Data":"cca792fc99fbfc8910bd748e63c5d3c3a5e219ff53a1ee5f8266e27a6e1dd184"} Oct 30 13:10:30.233813 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:30.233743 4503 scope.go:122] "RemoveContainer" containerID="734b1342f573163bd3b9b2272ce23bfd4b167dab866d3b422969fd8d63f75885" Oct 30 13:10:30.306776 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:30.306686 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7"] Oct 30 13:10:30.312462 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:30.312431 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-751d84fb-677d468bb5-479r7"] Oct 30 13:10:30.391520 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:30.391484 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a867e6d-42d0-4e78-bbd6-e4521a8e9b92" path="/var/lib/kubelet/pods/7a867e6d-42d0-4e78-bbd6-e4521a8e9b92/volumes" Oct 30 13:10:30.932829 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:30.932362 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm"] Oct 30 13:10:30.934419 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:30.933630 4503 cpu_manager.go:414] "containerMap: removing container" podUID="7a867e6d-42d0-4e78-bbd6-e4521a8e9b92" containerName="maas-api" Oct 30 13:10:30.934419 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:30.933657 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a867e6d-42d0-4e78-bbd6-e4521a8e9b92" containerName="maas-api" Oct 30 13:10:30.934419 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:30.933967 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a867e6d-42d0-4e78-bbd6-e4521a8e9b92" containerName="maas-api" Oct 30 13:10:31.010421 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.010379 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.019379 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.018029 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm"] Oct 30 13:10:31.031851 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.031801 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-0fd5fcba-gw-openshift-default-dockercfg-ltmlg\"" Oct 30 13:10:31.749352 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.749302 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:10:31.798040 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.762635 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-workload-certs\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.805453 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.805415 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-podinfo\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.815872 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.815833 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r29x\" (UniqueName: \"kubernetes.io/projected/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-kube-api-access-7r29x\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.826564 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.826525 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istiod-ca-cert\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.829154 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.829120 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-data\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.834395 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.789196 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:31.840226 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.834455 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-credential-socket\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.840604 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.840577 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-token\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.840891 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.840873 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-envoy\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.841082 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.841061 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-workload-socket\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.841438 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.841419 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:31.843292 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.843263 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:10:31.904580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.904531 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh" podUID="82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8" containerName="payload-processing" containerID="cri-o://676a786b6d485877477c2e0d93e1a0ab196944dc4048470140d6d0dc3738e77f" gracePeriod=25 Oct 30 13:10:31.943028 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.943002 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-envoy\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.943673 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.943651 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-workload-socket\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.943856 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.943844 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-workload-certs\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.943932 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.943922 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-podinfo\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.959259 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.959219 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r29x\" (UniqueName: \"kubernetes.io/projected/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-kube-api-access-7r29x\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.998003 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.977401 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istiod-ca-cert\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.998003 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.974746 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-workload-socket\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.998003 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.974524 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-workload-certs\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.998003 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.978612 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istiod-ca-cert\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.998003 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.995563 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-podinfo\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.998003 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.995818 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-envoy\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.998003 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.997003 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-data\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.998003 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.997045 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-data\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.998003 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.997153 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-credential-socket\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.998003 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.997194 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-token\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:31.998003 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:31.997868 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-credential-socket\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:32.000362 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:32.000300 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r29x\" (UniqueName: \"kubernetes.io/projected/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-kube-api-access-7r29x\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:32.022698 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:32.022660 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-token\") pod \"e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:32.298112 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:32.276869 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:33.229014 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:33.215383 4503 generic.go:356] "Generic (PLEG): container finished" podID="82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8" containerID="676a786b6d485877477c2e0d93e1a0ab196944dc4048470140d6d0dc3738e77f" exitCode=0 Oct 30 13:10:33.229014 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:33.220032 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh" event={"ID":"82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8","Type":"ContainerDied","Data":"676a786b6d485877477c2e0d93e1a0ab196944dc4048470140d6d0dc3738e77f"} Oct 30 13:10:33.372772 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:33.372443 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh" Oct 30 13:10:33.538646 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:33.538616 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8-plugins-config-volume" pod "82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8" (UID: "82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:33.543328 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:33.537283 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8-plugins-config-volume\") pod \"82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8\" (UID: \"82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8\") " Oct 30 13:10:33.543328 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:33.542190 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8-kube-api-access-6v8cj\" (UniqueName: \"kubernetes.io/projected/82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8-kube-api-access-6v8cj\") pod \"82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8\" (UID: \"82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8\") " Oct 30 13:10:33.548567 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:33.548519 4503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8-plugins-config-volume\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:33.558106 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:33.558069 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8-kube-api-access-6v8cj" pod "82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8" (UID: "82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8"). InnerVolumeSpecName "kube-api-access-6v8cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:33.649657 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:33.649626 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v8cj\" (UniqueName: \"kubernetes.io/projected/82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8-kube-api-access-6v8cj\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:33.758666 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:33.758569 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm"] Oct 30 13:10:33.764101 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:10:33.764067 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04fb46b3_a555_41ab_a7be_f57ed8c9deaf.slice/crio-5ff1eda75abc662105d27ab75182dfe90a592a628437c8ebd1378e4ffebb63f6 WatchSource:0}: Error finding container 5ff1eda75abc662105d27ab75182dfe90a592a628437c8ebd1378e4ffebb63f6: Status 404 returned error can't find the container with id 5ff1eda75abc662105d27ab75182dfe90a592a628437c8ebd1378e4ffebb63f6 Oct 30 13:10:33.767596 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:33.767550 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:33.767694 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:33.767675 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:33.767758 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:33.767722 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:34.318975 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:34.318063 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" event={"ID":"04fb46b3-a555-41ab-a7be-f57ed8c9deaf","Type":"ContainerStarted","Data":"f024dbffe0f78e624a039c3ffe55080f2d5035d2d6b6b54088feeed6d5d78a96"} Oct 30 13:10:34.318975 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:34.318151 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" event={"ID":"04fb46b3-a555-41ab-a7be-f57ed8c9deaf","Type":"ContainerStarted","Data":"5ff1eda75abc662105d27ab75182dfe90a592a628437c8ebd1378e4ffebb63f6"} Oct 30 13:10:34.326961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:34.325565 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh" Oct 30 13:10:34.332046 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:34.332004 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh" event={"ID":"82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8","Type":"ContainerDied","Data":"78f4e33393f175b21e754e650c65984c75138c8506cd4511cacfddb9c8c5039c"} Oct 30 13:10:34.332321 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:34.332286 4503 scope.go:122] "RemoveContainer" containerID="676a786b6d485877477c2e0d93e1a0ab196944dc4048470140d6d0dc3738e77f" Oct 30 13:10:34.573900 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:34.573760 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" podStartSLOduration=4.573738599 podStartE2EDuration="4.573738599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:34.389546576 +0000 UTC m=+1128.212774865" watchObservedRunningTime="2026-10-30 13:10:34.573738599 +0000 UTC m=+1128.396966876" Oct 30 13:10:34.736462 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:34.736404 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh"] Oct 30 13:10:34.843022 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:34.836075 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4a59cbc1-675dd985bd-6qhlh"] Oct 30 13:10:35.303963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:35.298427 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:35.303963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:35.298566 4503 patch_prober.go:29] interesting pod/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 13:10:35.303963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:35.298613 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" podUID="04fb46b3-a555-41ab-a7be-f57ed8c9deaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 13:10:35.680961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:35.674673 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv"] Oct 30 13:10:35.680961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:35.675823 4503 cpu_manager.go:414] "containerMap: removing container" podUID="82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8" containerName="payload-processing" Oct 30 13:10:35.680961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:35.675864 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8" containerName="payload-processing" Oct 30 13:10:35.680961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:35.676141 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8" containerName="payload-processing" Oct 30 13:10:35.680961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:35.680476 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:35.725534 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:35.725497 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv"] Oct 30 13:10:35.838853 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:35.812595 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0332beb-0b14-493b-9a37-4de43236c083-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.433713 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.428673 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.433713 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.428953 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.433713 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.429014 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.433713 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.429185 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0332beb-0b14-493b-9a37-4de43236c083-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.433713 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.429272 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0332beb-0b14-493b-9a37-4de43236c083-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.433713 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.429345 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.433713 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.429401 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm6qd\" (UniqueName: \"kubernetes.io/projected/e0332beb-0b14-493b-9a37-4de43236c083-kube-api-access-rm6qd\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.433713 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.429443 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.576534 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.576271 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.624835 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.624791 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.625049 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.625023 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0332beb-0b14-493b-9a37-4de43236c083-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.625135 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.625116 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0332beb-0b14-493b-9a37-4de43236c083-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.625206 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.625192 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.625273 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.625246 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm6qd\" (UniqueName: \"kubernetes.io/projected/e0332beb-0b14-493b-9a37-4de43236c083-kube-api-access-rm6qd\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.625347 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.625308 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.625396 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.625368 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0332beb-0b14-493b-9a37-4de43236c083-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.625450 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.625409 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.626902 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.626065 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-credential-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.626902 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.603786 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-workload-certs\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.692023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.680639 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0332beb-0b14-493b-9a37-4de43236c083-istiod-ca-cert\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.724099 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.716472 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-istio-data\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.724099 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.717092 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-workload-socket\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.816755 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.812632 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-istio-envoy\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.816755 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.813062 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:36.816755 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.814142 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0332beb-0b14-493b-9a37-4de43236c083-istio-token\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.820258 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.819734 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0332beb-0b14-493b-9a37-4de43236c083-istio-podinfo\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.917158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.830789 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm6qd\" (UniqueName: \"kubernetes.io/projected/e0332beb-0b14-493b-9a37-4de43236c083-kube-api-access-rm6qd\") pod \"e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.968382 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.968291 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:36.968697 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.968659 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8" path="/var/lib/kubelet/pods/82f6e537-8a1b-42bc-9462-a3b5d5cbdcf8/volumes" Oct 30 13:10:36.969158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.969136 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:36.969267 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:36.969187 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:37.410430 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:37.410402 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv"] Oct 30 13:10:37.410997 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:10:37.410929 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0332beb_0b14_493b_9a37_4de43236c083.slice/crio-b658df5da5752e614cf08cb6a69b9e8075096b9c859c62e84b2495ef901eb9e0 WatchSource:0}: Error finding container b658df5da5752e614cf08cb6a69b9e8075096b9c859c62e84b2495ef901eb9e0: Status 404 returned error can't find the container with id b658df5da5752e614cf08cb6a69b9e8075096b9c859c62e84b2495ef901eb9e0 Oct 30 13:10:37.413867 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:37.413832 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:37.413980 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:37.413897 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:37.413980 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:37.413924 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:37.830863 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:37.830761 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" event={"ID":"e0332beb-0b14-493b-9a37-4de43236c083","Type":"ContainerStarted","Data":"b71c4743a919c38ffd47173db043f957593e20052b590a28054cf60977388916"} Oct 30 13:10:37.830863 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:37.830826 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" event={"ID":"e0332beb-0b14-493b-9a37-4de43236c083","Type":"ContainerStarted","Data":"b658df5da5752e614cf08cb6a69b9e8075096b9c859c62e84b2495ef901eb9e0"} Oct 30 13:10:37.832656 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:37.832632 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:10:37.899199 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:37.899129 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" podStartSLOduration=2.899109363 podStartE2EDuration="2.899109363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:37.895076046 +0000 UTC m=+1131.718304339" watchObservedRunningTime="2026-10-30 13:10:37.899109363 +0000 UTC m=+1131.722337642" Oct 30 13:10:38.013290 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:38.011433 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:38.013290 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:38.011561 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:10:38.013290 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:38.011728 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" podUID="e0332beb-0b14-493b-9a37-4de43236c083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 13:10:38.996963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:38.996298 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:10:38.996963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:38.996359 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" podUID="e0332beb-0b14-493b-9a37-4de43236c083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 13:10:39.343569 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.342568 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:10:39.430676 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.430624 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-envoy\") pod \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " Oct 30 13:10:39.451216 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.451173 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-workload-socket\") pod \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " Oct 30 13:10:39.451746 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.451706 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-credential-socket\") pod \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " Oct 30 13:10:39.467196 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.467148 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-workload-certs" pod "1cc2d750-6f71-4f16-8228-82e6390d9bcf" (UID: "1cc2d750-6f71-4f16-8228-82e6390d9bcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:39.474809 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.474775 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-workload-certs\") pod \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " Oct 30 13:10:39.477212 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.477180 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-data\") pod \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " Oct 30 13:10:39.481185 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.481159 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istiod-ca-cert\") pod \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " Oct 30 13:10:39.481410 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.481391 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-token\" (UniqueName: \"kubernetes.io/projected/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-token\") pod \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " Oct 30 13:10:39.526089 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.510341 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istiod-ca-cert" pod "1cc2d750-6f71-4f16-8228-82e6390d9bcf" (UID: "1cc2d750-6f71-4f16-8228-82e6390d9bcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:39.526089 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.510681 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-data" pod "1cc2d750-6f71-4f16-8228-82e6390d9bcf" (UID: "1cc2d750-6f71-4f16-8228-82e6390d9bcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:39.566848 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.564072 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-workload-socket" pod "1cc2d750-6f71-4f16-8228-82e6390d9bcf" (UID: "1cc2d750-6f71-4f16-8228-82e6390d9bcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:39.645903 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.645865 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-podinfo" pod "1cc2d750-6f71-4f16-8228-82e6390d9bcf" (UID: "1cc2d750-6f71-4f16-8228-82e6390d9bcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:10:39.684605 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.678579 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-credential-socket" pod "1cc2d750-6f71-4f16-8228-82e6390d9bcf" (UID: "1cc2d750-6f71-4f16-8228-82e6390d9bcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:39.684746 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.481542 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-podinfo\") pod \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " Oct 30 13:10:39.685039 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.685014 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cc2d750-6f71-4f16-8228-82e6390d9bcf-kube-api-access-kr5c6\" (UniqueName: \"kubernetes.io/projected/1cc2d750-6f71-4f16-8228-82e6390d9bcf-kube-api-access-kr5c6\") pod \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\" (UID: \"1cc2d750-6f71-4f16-8228-82e6390d9bcf\") " Oct 30 13:10:39.685495 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.685472 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-envoy" pod "1cc2d750-6f71-4f16-8228-82e6390d9bcf" (UID: "1cc2d750-6f71-4f16-8228-82e6390d9bcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:39.686383 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.686360 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:39.696163 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.696133 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:39.696326 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.696311 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:39.696453 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.696437 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:39.696572 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.696560 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:39.696689 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.696676 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:39.696988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.696793 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc2d750-6f71-4f16-8228-82e6390d9bcf-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:39.717165 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.717111 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc2d750-6f71-4f16-8228-82e6390d9bcf-kube-api-access-kr5c6" pod "1cc2d750-6f71-4f16-8228-82e6390d9bcf" (UID: "1cc2d750-6f71-4f16-8228-82e6390d9bcf"). InnerVolumeSpecName "kube-api-access-kr5c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:39.736390 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.736348 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-token" pod "1cc2d750-6f71-4f16-8228-82e6390d9bcf" (UID: "1cc2d750-6f71-4f16-8228-82e6390d9bcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:39.816624 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.815676 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr5c6\" (UniqueName: \"kubernetes.io/projected/1cc2d750-6f71-4f16-8228-82e6390d9bcf-kube-api-access-kr5c6\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:39.816624 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.815712 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc2d750-6f71-4f16-8228-82e6390d9bcf-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:39.938367 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.938322 4503 generic.go:356] "Generic (PLEG): container finished" podID="1cc2d750-6f71-4f16-8228-82e6390d9bcf" containerID="7f3d4ae25203dd801808fbc38b71461639397b9ed2b3029e17623a36fcbb549b" exitCode=0 Oct 30 13:10:39.938729 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.938700 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" event={"ID":"1cc2d750-6f71-4f16-8228-82e6390d9bcf","Type":"ContainerDied","Data":"7f3d4ae25203dd801808fbc38b71461639397b9ed2b3029e17623a36fcbb549b"} Oct 30 13:10:39.938981 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.938962 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" event={"ID":"1cc2d750-6f71-4f16-8228-82e6390d9bcf","Type":"ContainerDied","Data":"b0041d79478b8330d1a988ca474707503b7d1ff20461210a4313e0854248f008"} Oct 30 13:10:39.939124 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.939111 4503 scope.go:122] "RemoveContainer" containerID="7f3d4ae25203dd801808fbc38b71461639397b9ed2b3029e17623a36fcbb549b" Oct 30 13:10:39.939322 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.938825 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j" Oct 30 13:10:39.982965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.982438 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:10:39.982965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:39.982509 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" podUID="e0332beb-0b14-493b-9a37-4de43236c083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 13:10:40.019706 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:40.019667 4503 scope.go:122] "RemoveContainer" containerID="a0bb50ffd774b138f89e420793c04af18f1c7fb10cd349438c0ced1c6eeb1c25" Oct 30 13:10:40.033219 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:40.033191 4503 scope.go:122] "RemoveContainer" containerID="7f3d4ae25203dd801808fbc38b71461639397b9ed2b3029e17623a36fcbb549b" Oct 30 13:10:40.033517 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:10:40.033497 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f3d4ae25203dd801808fbc38b71461639397b9ed2b3029e17623a36fcbb549b\": container with ID starting with 7f3d4ae25203dd801808fbc38b71461639397b9ed2b3029e17623a36fcbb549b not found: ID does not exist" containerID="7f3d4ae25203dd801808fbc38b71461639397b9ed2b3029e17623a36fcbb549b" Oct 30 13:10:40.033571 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:40.033526 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3d4ae25203dd801808fbc38b71461639397b9ed2b3029e17623a36fcbb549b"} err="failed to get container status \"7f3d4ae25203dd801808fbc38b71461639397b9ed2b3029e17623a36fcbb549b\": rpc error: code = NotFound desc = could not find container \"7f3d4ae25203dd801808fbc38b71461639397b9ed2b3029e17623a36fcbb549b\": container with ID starting with 7f3d4ae25203dd801808fbc38b71461639397b9ed2b3029e17623a36fcbb549b not found: ID does not exist" Oct 30 13:10:40.033571 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:40.033543 4503 scope.go:122] "RemoveContainer" containerID="a0bb50ffd774b138f89e420793c04af18f1c7fb10cd349438c0ced1c6eeb1c25" Oct 30 13:10:40.033768 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:10:40.033753 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0bb50ffd774b138f89e420793c04af18f1c7fb10cd349438c0ced1c6eeb1c25\": container with ID starting with a0bb50ffd774b138f89e420793c04af18f1c7fb10cd349438c0ced1c6eeb1c25 not found: ID does not exist" containerID="a0bb50ffd774b138f89e420793c04af18f1c7fb10cd349438c0ced1c6eeb1c25" Oct 30 13:10:40.033811 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:40.033771 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bb50ffd774b138f89e420793c04af18f1c7fb10cd349438c0ced1c6eeb1c25"} err="failed to get container status \"a0bb50ffd774b138f89e420793c04af18f1c7fb10cd349438c0ced1c6eeb1c25\": rpc error: code = NotFound desc = could not find container \"a0bb50ffd774b138f89e420793c04af18f1c7fb10cd349438c0ced1c6eeb1c25\": container with ID starting with a0bb50ffd774b138f89e420793c04af18f1c7fb10cd349438c0ced1c6eeb1c25 not found: ID does not exist" Oct 30 13:10:40.111754 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:40.111608 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j"] Oct 30 13:10:40.120042 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:40.120009 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-751d84fb-openshift-default-74d9499f8c-qkw7j"] Oct 30 13:10:40.403019 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:40.398616 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cc2d750-6f71-4f16-8228-82e6390d9bcf" path="/var/lib/kubelet/pods/1cc2d750-6f71-4f16-8228-82e6390d9bcf/volumes" Oct 30 13:10:40.990968 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:40.983095 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:10:40.990968 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:40.983172 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" podUID="e0332beb-0b14-493b-9a37-4de43236c083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 13:10:41.789563 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:41.789523 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv"] Oct 30 13:10:41.791635 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:41.791598 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" podUID="04a3e261-7af6-44ae-8449-bbc8d016f239" containerName="maas-api" containerID="cri-o://972201b292958fa2e9d5dcdd0f63513375c1e1e2b3295432d52d766005f068e5" gracePeriod=30 Oct 30 13:10:41.904178 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:41.903350 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" podUID="04a3e261-7af6-44ae-8449-bbc8d016f239" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.87:8443/health\": dial tcp 10.133.0.87:8443: connect: connection refused" Oct 30 13:10:41.984836 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:41.984358 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:10:41.984836 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:41.984438 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" podUID="e0332beb-0b14-493b-9a37-4de43236c083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 13:10:42.685966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:42.685356 4503 generic.go:356] "Generic (PLEG): container finished" podID="04a3e261-7af6-44ae-8449-bbc8d016f239" containerID="972201b292958fa2e9d5dcdd0f63513375c1e1e2b3295432d52d766005f068e5" exitCode=0 Oct 30 13:10:42.721975 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:42.721917 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" event={"ID":"04a3e261-7af6-44ae-8449-bbc8d016f239","Type":"ContainerDied","Data":"972201b292958fa2e9d5dcdd0f63513375c1e1e2b3295432d52d766005f068e5"} Oct 30 13:10:42.825737 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:42.825357 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" Oct 30 13:10:42.938103 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:42.912605 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a3e261-7af6-44ae-8449-bbc8d016f239-kube-api-access-dwmwg\" (UniqueName: \"kubernetes.io/projected/04a3e261-7af6-44ae-8449-bbc8d016f239-kube-api-access-dwmwg\") pod \"04a3e261-7af6-44ae-8449-bbc8d016f239\" (UID: \"04a3e261-7af6-44ae-8449-bbc8d016f239\") " Oct 30 13:10:42.938103 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:42.912686 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a3e261-7af6-44ae-8449-bbc8d016f239-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a3e261-7af6-44ae-8449-bbc8d016f239-openshift-service-ca\") pod \"04a3e261-7af6-44ae-8449-bbc8d016f239\" (UID: \"04a3e261-7af6-44ae-8449-bbc8d016f239\") " Oct 30 13:10:42.938103 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:42.912784 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04a3e261-7af6-44ae-8449-bbc8d016f239-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04a3e261-7af6-44ae-8449-bbc8d016f239-metrics-certs\") pod \"04a3e261-7af6-44ae-8449-bbc8d016f239\" (UID: \"04a3e261-7af6-44ae-8449-bbc8d016f239\") " Oct 30 13:10:42.938103 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:42.912877 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04a3e261-7af6-44ae-8449-bbc8d016f239-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a3e261-7af6-44ae-8449-bbc8d016f239-maas-api-tls\") pod \"04a3e261-7af6-44ae-8449-bbc8d016f239\" (UID: \"04a3e261-7af6-44ae-8449-bbc8d016f239\") " Oct 30 13:10:42.976160 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:42.958867 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a3e261-7af6-44ae-8449-bbc8d016f239-openshift-service-ca" pod "04a3e261-7af6-44ae-8449-bbc8d016f239" (UID: "04a3e261-7af6-44ae-8449-bbc8d016f239"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:42.981731 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:42.981482 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:10:42.981731 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:42.981548 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" podUID="e0332beb-0b14-493b-9a37-4de43236c083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 13:10:43.048788 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:43.041432 4503 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a3e261-7af6-44ae-8449-bbc8d016f239-openshift-service-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:43.151970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:43.135519 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a3e261-7af6-44ae-8449-bbc8d016f239-metrics-certs" pod "04a3e261-7af6-44ae-8449-bbc8d016f239" (UID: "04a3e261-7af6-44ae-8449-bbc8d016f239"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:43.151970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:43.135664 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a3e261-7af6-44ae-8449-bbc8d016f239-maas-api-tls" pod "04a3e261-7af6-44ae-8449-bbc8d016f239" (UID: "04a3e261-7af6-44ae-8449-bbc8d016f239"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:43.173245 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:43.170127 4503 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04a3e261-7af6-44ae-8449-bbc8d016f239-metrics-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:43.173245 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:43.170161 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a3e261-7af6-44ae-8449-bbc8d016f239-maas-api-tls\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:43.206464 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:43.205211 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a3e261-7af6-44ae-8449-bbc8d016f239-kube-api-access-dwmwg" pod "04a3e261-7af6-44ae-8449-bbc8d016f239" (UID: "04a3e261-7af6-44ae-8449-bbc8d016f239"). InnerVolumeSpecName "kube-api-access-dwmwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:43.283965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:43.283051 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwmwg\" (UniqueName: \"kubernetes.io/projected/04a3e261-7af6-44ae-8449-bbc8d016f239-kube-api-access-dwmwg\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:44.002968 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:43.981830 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:10:44.018005 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:44.008253 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" podUID="e0332beb-0b14-493b-9a37-4de43236c083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 13:10:44.018005 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:43.985131 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" Oct 30 13:10:44.018005 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:43.985026 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv" event={"ID":"04a3e261-7af6-44ae-8449-bbc8d016f239","Type":"ContainerDied","Data":"07290b5f7c686a25bce4bcfb3ba441db5fc5e0d262e71119915677457b059da2"} Oct 30 13:10:44.018005 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:44.010825 4503 scope.go:122] "RemoveContainer" containerID="972201b292958fa2e9d5dcdd0f63513375c1e1e2b3295432d52d766005f068e5" Oct 30 13:10:44.287240 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:44.277987 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv"] Oct 30 13:10:44.293064 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:44.291690 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4a59cbc1-dfc9d889f-ndmdv"] Oct 30 13:10:44.392357 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:44.392024 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a3e261-7af6-44ae-8449-bbc8d016f239" path="/var/lib/kubelet/pods/04a3e261-7af6-44ae-8449-bbc8d016f239/volumes" Oct 30 13:10:44.563468 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:44.563070 4503 patch_prober.go:29] interesting pod/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:10:44.563468 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:44.563136 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" podUID="0549ac29-5939-4f6c-aced-091b158f4afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:44.969973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:44.969342 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:10:44.969973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:44.969406 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" podUID="e0332beb-0b14-493b-9a37-4de43236c083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 13:10:45.974964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:45.972722 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:10:45.974964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:45.972785 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" podUID="e0332beb-0b14-493b-9a37-4de43236c083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 13:10:46.882794 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:46.882761 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b"] Oct 30 13:10:46.976919 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:46.976889 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:46.978650 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:46.978591 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:46.979795 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:46.979748 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:10:49.064011 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:49.063867 4503 generic.go:356] "Generic (PLEG): container finished" podID="0549ac29-5939-4f6c-aced-091b158f4afd" containerID="bdd197d00bc095e8f4d4a0a9a86e5c64e90b853d648b65df3aea523182e18ea8" exitCode=0 Oct 30 13:10:49.064011 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:49.063980 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" event={"ID":"0549ac29-5939-4f6c-aced-091b158f4afd","Type":"ContainerDied","Data":"bdd197d00bc095e8f4d4a0a9a86e5c64e90b853d648b65df3aea523182e18ea8"} Oct 30 13:10:49.064580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:49.064562 4503 scope.go:122] "RemoveContainer" containerID="bdd197d00bc095e8f4d4a0a9a86e5c64e90b853d648b65df3aea523182e18ea8" Oct 30 13:10:49.066755 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:49.066716 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:49.066857 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:49.066807 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:49.066903 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:49.066854 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:50.084903 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:50.083171 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" event={"ID":"0549ac29-5939-4f6c-aced-091b158f4afd","Type":"ContainerStarted","Data":"f65cd9f96200812784e430e43dc23a011dd406b61d3d2f98c4a952aa3149b619"} Oct 30 13:10:50.276159 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:50.276083 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:10:50.278982 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:50.276828 4503 patch_prober.go:29] interesting pod/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 13:10:50.278982 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:50.276900 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" podUID="0549ac29-5939-4f6c-aced-091b158f4afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 13:10:50.343242 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:50.343162 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv"] Oct 30 13:10:51.285282 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:51.284457 4503 patch_prober.go:29] interesting pod/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 13:10:51.285282 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:51.284515 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" podUID="0549ac29-5939-4f6c-aced-091b158f4afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 13:10:51.884718 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:51.884630 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b" podUID="08339a18-3dd6-4f65-bff1-3c3b83d8ea5b" containerName="payload-pre-processing" containerID="cri-o://acde95ca3287bb0a15e1a6ee98df3a71a124293e5d74bda8e52d243da35a0fe7" gracePeriod=25 Oct 30 13:10:52.217965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:52.217020 4503 generic.go:356] "Generic (PLEG): container finished" podID="08339a18-3dd6-4f65-bff1-3c3b83d8ea5b" containerID="acde95ca3287bb0a15e1a6ee98df3a71a124293e5d74bda8e52d243da35a0fe7" exitCode=0 Oct 30 13:10:52.225964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:52.218740 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b" event={"ID":"08339a18-3dd6-4f65-bff1-3c3b83d8ea5b","Type":"ContainerDied","Data":"acde95ca3287bb0a15e1a6ee98df3a71a124293e5d74bda8e52d243da35a0fe7"} Oct 30 13:10:52.289965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:52.285613 4503 patch_prober.go:29] interesting pod/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 13:10:52.289965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:52.285682 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" podUID="0549ac29-5939-4f6c-aced-091b158f4afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 13:10:53.036211 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:53.036181 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b" Oct 30 13:10:53.307022 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:53.284932 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08339a18-3dd6-4f65-bff1-3c3b83d8ea5b-kube-api-access-fr6pb\" (UniqueName: \"kubernetes.io/projected/08339a18-3dd6-4f65-bff1-3c3b83d8ea5b-kube-api-access-fr6pb\") pod \"08339a18-3dd6-4f65-bff1-3c3b83d8ea5b\" (UID: \"08339a18-3dd6-4f65-bff1-3c3b83d8ea5b\") " Oct 30 13:10:53.307022 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:53.285128 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08339a18-3dd6-4f65-bff1-3c3b83d8ea5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08339a18-3dd6-4f65-bff1-3c3b83d8ea5b-plugins-config-volume\") pod \"08339a18-3dd6-4f65-bff1-3c3b83d8ea5b\" (UID: \"08339a18-3dd6-4f65-bff1-3c3b83d8ea5b\") " Oct 30 13:10:53.307022 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:53.299921 4503 patch_prober.go:29] interesting pod/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 13:10:53.307022 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:53.299997 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" podUID="0549ac29-5939-4f6c-aced-091b158f4afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 13:10:53.307980 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:53.307922 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08339a18-3dd6-4f65-bff1-3c3b83d8ea5b-plugins-config-volume" pod "08339a18-3dd6-4f65-bff1-3c3b83d8ea5b" (UID: "08339a18-3dd6-4f65-bff1-3c3b83d8ea5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:53.318408 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:53.318376 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b" event={"ID":"08339a18-3dd6-4f65-bff1-3c3b83d8ea5b","Type":"ContainerDied","Data":"439f3d27451fa49593367a553d716caee98917bf1a54e78e9825f5ac45a8da71"} Oct 30 13:10:53.319035 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:53.319005 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08339a18-3dd6-4f65-bff1-3c3b83d8ea5b-kube-api-access-fr6pb" pod "08339a18-3dd6-4f65-bff1-3c3b83d8ea5b" (UID: "08339a18-3dd6-4f65-bff1-3c3b83d8ea5b"). InnerVolumeSpecName "kube-api-access-fr6pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:53.319155 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:53.319013 4503 scope.go:122] "RemoveContainer" containerID="acde95ca3287bb0a15e1a6ee98df3a71a124293e5d74bda8e52d243da35a0fe7" Oct 30 13:10:53.319155 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:53.318394 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b" Oct 30 13:10:53.427142 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:53.391592 4503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08339a18-3dd6-4f65-bff1-3c3b83d8ea5b-plugins-config-volume\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:53.427142 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:53.391634 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr6pb\" (UniqueName: \"kubernetes.io/projected/08339a18-3dd6-4f65-bff1-3c3b83d8ea5b-kube-api-access-fr6pb\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:53.427607 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:53.427352 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b"] Oct 30 13:10:53.431650 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:53.431619 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-253f7c37-5f55b58875-rxp4b"] Oct 30 13:10:53.922903 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:53.921926 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z"] Oct 30 13:10:53.922903 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:53.922236 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" podUID="acd6e615-1e54-4058-b289-a1fa783e00a7" containerName="maas-api" containerID="cri-o://f3e1619648b759e9ba0cd014ca13f611108a35d159beca27270c41eda764798f" gracePeriod=30 Oct 30 13:10:54.268026 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:54.252072 4503 patch_prober.go:29] interesting pod/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 13:10:54.289095 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:54.268553 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" podUID="0549ac29-5939-4f6c-aced-091b158f4afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 13:10:54.557029 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:54.554143 4503 generic.go:356] "Generic (PLEG): container finished" podID="acd6e615-1e54-4058-b289-a1fa783e00a7" containerID="f3e1619648b759e9ba0cd014ca13f611108a35d159beca27270c41eda764798f" exitCode=0 Oct 30 13:10:54.966125 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:54.926358 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08339a18-3dd6-4f65-bff1-3c3b83d8ea5b" path="/var/lib/kubelet/pods/08339a18-3dd6-4f65-bff1-3c3b83d8ea5b/volumes" Oct 30 13:10:54.966125 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:54.926847 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" event={"ID":"acd6e615-1e54-4058-b289-a1fa783e00a7","Type":"ContainerDied","Data":"f3e1619648b759e9ba0cd014ca13f611108a35d159beca27270c41eda764798f"} Oct 30 13:10:55.138177 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.138142 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" Oct 30 13:10:55.150091 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.141569 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acd6e615-1e54-4058-b289-a1fa783e00a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acd6e615-1e54-4058-b289-a1fa783e00a7-maas-api-tls\") pod \"acd6e615-1e54-4058-b289-a1fa783e00a7\" (UID: \"acd6e615-1e54-4058-b289-a1fa783e00a7\") " Oct 30 13:10:55.150091 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.141707 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd6e615-1e54-4058-b289-a1fa783e00a7-kube-api-access-m4h7n\" (UniqueName: \"kubernetes.io/projected/acd6e615-1e54-4058-b289-a1fa783e00a7-kube-api-access-m4h7n\") pod \"acd6e615-1e54-4058-b289-a1fa783e00a7\" (UID: \"acd6e615-1e54-4058-b289-a1fa783e00a7\") " Oct 30 13:10:55.150091 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.141808 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acd6e615-1e54-4058-b289-a1fa783e00a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd6e615-1e54-4058-b289-a1fa783e00a7-openshift-service-ca\") pod \"acd6e615-1e54-4058-b289-a1fa783e00a7\" (UID: \"acd6e615-1e54-4058-b289-a1fa783e00a7\") " Oct 30 13:10:55.150091 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.141893 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acd6e615-1e54-4058-b289-a1fa783e00a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/acd6e615-1e54-4058-b289-a1fa783e00a7-metrics-certs\") pod \"acd6e615-1e54-4058-b289-a1fa783e00a7\" (UID: \"acd6e615-1e54-4058-b289-a1fa783e00a7\") " Oct 30 13:10:55.204965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.153827 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd6e615-1e54-4058-b289-a1fa783e00a7-openshift-service-ca" pod "acd6e615-1e54-4058-b289-a1fa783e00a7" (UID: "acd6e615-1e54-4058-b289-a1fa783e00a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:55.204965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.177044 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acd6e615-1e54-4058-b289-a1fa783e00a7-maas-api-tls" pod "acd6e615-1e54-4058-b289-a1fa783e00a7" (UID: "acd6e615-1e54-4058-b289-a1fa783e00a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:55.223243 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.212113 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd6e615-1e54-4058-b289-a1fa783e00a7-kube-api-access-m4h7n" pod "acd6e615-1e54-4058-b289-a1fa783e00a7" (UID: "acd6e615-1e54-4058-b289-a1fa783e00a7"). InnerVolumeSpecName "kube-api-access-m4h7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:55.244225 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.244185 4503 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd6e615-1e54-4058-b289-a1fa783e00a7-openshift-service-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:55.244521 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.244498 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acd6e615-1e54-4058-b289-a1fa783e00a7-maas-api-tls\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:55.244645 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.244629 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4h7n\" (UniqueName: \"kubernetes.io/projected/acd6e615-1e54-4058-b289-a1fa783e00a7-kube-api-access-m4h7n\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:55.265759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.265078 4503 patch_prober.go:29] interesting pod/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 13:10:55.306970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.266035 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" podUID="0549ac29-5939-4f6c-aced-091b158f4afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 13:10:55.306970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.294965 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acd6e615-1e54-4058-b289-a1fa783e00a7-metrics-certs" pod "acd6e615-1e54-4058-b289-a1fa783e00a7" (UID: "acd6e615-1e54-4058-b289-a1fa783e00a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:55.349103 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.349037 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" podUID="803ac7a6-439b-4634-a116-5e16c2595f42" containerName="payload-pre-processing" containerID="cri-o://6867ab281b6bf64d6dfb629e3c907371b7390cad875cc8e85e44ca3427e9afac" gracePeriod=25 Oct 30 13:10:55.374243 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.371001 4503 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acd6e615-1e54-4058-b289-a1fa783e00a7-metrics-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:10:55.565474 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.565377 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx"] Oct 30 13:10:55.566383 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.566327 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" podUID="8daeac71-175e-4684-84df-b27b0d52e733" containerName="istio-proxy" containerID="cri-o://2a3dbc6b593fa6b5f6af8b9510907704a87be349f9a8c5fd2219811b1ab2faf0" gracePeriod=30 Oct 30 13:10:55.616220 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.616171 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" event={"ID":"acd6e615-1e54-4058-b289-a1fa783e00a7","Type":"ContainerDied","Data":"1f9b66d8965a01ca33f91d38e5090980e064042dfb752c9de6cebf28eb42e5ad"} Oct 30 13:10:55.616532 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.616512 4503 scope.go:122] "RemoveContainer" containerID="f3e1619648b759e9ba0cd014ca13f611108a35d159beca27270c41eda764798f" Oct 30 13:10:55.616875 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:55.616858 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z" Oct 30 13:10:56.272043 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:56.261824 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd"] Oct 30 13:10:56.272043 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:56.262143 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" podUID="d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" containerName="istio-proxy" containerID="cri-o://994775eeb5e92033429ace2d3d4fdf0d07b3edfa3e9be5b169f386c682316634" gracePeriod=30 Oct 30 13:10:56.287553 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:56.279372 4503 patch_prober.go:29] interesting pod/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 13:10:56.287553 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:56.279442 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" podUID="0549ac29-5939-4f6c-aced-091b158f4afd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 13:10:56.441993 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:56.439468 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4"] Oct 30 13:10:56.441993 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:56.439560 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6"] Oct 30 13:10:56.441993 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:56.439761 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" podUID="dc8ab9ac-098f-4b41-9f0d-51422627d481" containerName="istio-proxy" containerID="cri-o://0f240e34b87ea0291df9fef93f44dfc807ce3b33cbd42f0f63857773ce414673" gracePeriod=30 Oct 30 13:10:56.441993 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:56.440047 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" podUID="bec8b24c-a530-4514-9f0b-5a11af7776ce" containerName="istio-proxy" containerID="cri-o://7398c73c609808de1e473d0405e9e5c221e59ad2e2c2cd09544fd166387362e3" gracePeriod=30 Oct 30 13:10:56.464908 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:56.464862 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z"] Oct 30 13:10:56.469919 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:56.469867 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-253f7c37-5fbbd9c6-hhc4z"] Oct 30 13:10:56.999136 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:56.993154 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2"] Oct 30 13:10:56.999136 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:56.993482 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" podUID="0549ac29-5939-4f6c-aced-091b158f4afd" containerName="istio-proxy" containerID="cri-o://f65cd9f96200812784e430e43dc23a011dd406b61d3d2f98c4a952aa3149b619" gracePeriod=30 Oct 30 13:10:58.365173 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.365141 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acd6e615-1e54-4058-b289-a1fa783e00a7" path="/var/lib/kubelet/pods/acd6e615-1e54-4058-b289-a1fa783e00a7/volumes" Oct 30 13:10:58.439772 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.439732 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x"] Oct 30 13:10:58.440789 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.440770 4503 cpu_manager.go:414] "containerMap: removing container" podUID="1cc2d750-6f71-4f16-8228-82e6390d9bcf" containerName="istio-proxy" Oct 30 13:10:58.440853 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.440791 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cc2d750-6f71-4f16-8228-82e6390d9bcf" containerName="istio-proxy" Oct 30 13:10:58.440853 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.440810 4503 cpu_manager.go:414] "containerMap: removing container" podUID="1cc2d750-6f71-4f16-8228-82e6390d9bcf" containerName="istio-proxy" Oct 30 13:10:58.440853 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.440818 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cc2d750-6f71-4f16-8228-82e6390d9bcf" containerName="istio-proxy" Oct 30 13:10:58.440853 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.440834 4503 cpu_manager.go:414] "containerMap: removing container" podUID="04a3e261-7af6-44ae-8449-bbc8d016f239" containerName="maas-api" Oct 30 13:10:58.440853 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.440845 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a3e261-7af6-44ae-8449-bbc8d016f239" containerName="maas-api" Oct 30 13:10:58.441043 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.440869 4503 cpu_manager.go:414] "containerMap: removing container" podUID="08339a18-3dd6-4f65-bff1-3c3b83d8ea5b" containerName="payload-pre-processing" Oct 30 13:10:58.441043 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.440877 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08339a18-3dd6-4f65-bff1-3c3b83d8ea5b" containerName="payload-pre-processing" Oct 30 13:10:58.441043 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.440920 4503 cpu_manager.go:414] "containerMap: removing container" podUID="acd6e615-1e54-4058-b289-a1fa783e00a7" containerName="maas-api" Oct 30 13:10:58.441043 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.440927 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd6e615-1e54-4058-b289-a1fa783e00a7" containerName="maas-api" Oct 30 13:10:58.441157 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.441102 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="08339a18-3dd6-4f65-bff1-3c3b83d8ea5b" containerName="payload-pre-processing" Oct 30 13:10:58.441157 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.441120 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd6e615-1e54-4058-b289-a1fa783e00a7" containerName="maas-api" Oct 30 13:10:58.441157 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.441136 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cc2d750-6f71-4f16-8228-82e6390d9bcf" containerName="istio-proxy" Oct 30 13:10:58.441157 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.441145 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cc2d750-6f71-4f16-8228-82e6390d9bcf" containerName="istio-proxy" Oct 30 13:10:58.441157 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.441153 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a3e261-7af6-44ae-8449-bbc8d016f239" containerName="maas-api" Oct 30 13:10:58.444890 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.444868 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" Oct 30 13:10:58.469343 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.469310 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x"] Oct 30 13:10:58.474378 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.474349 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-0fd5fcba-serving-cert\"" Oct 30 13:10:58.477502 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.477474 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:10:58.547250 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.547208 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk6m7\" (UniqueName: \"kubernetes.io/projected/391723a7-7e4d-4727-af52-abe87e1ab8ff-kube-api-access-qk6m7\") pod \"maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x\" (UID: \"391723a7-7e4d-4727-af52-abe87e1ab8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" Oct 30 13:10:58.547454 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.547279 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/391723a7-7e4d-4727-af52-abe87e1ab8ff-maas-api-tls\") pod \"maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x\" (UID: \"391723a7-7e4d-4727-af52-abe87e1ab8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" Oct 30 13:10:58.547577 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.547549 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/391723a7-7e4d-4727-af52-abe87e1ab8ff-metrics-certs\") pod \"maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x\" (UID: \"391723a7-7e4d-4727-af52-abe87e1ab8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" Oct 30 13:10:58.547708 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.547687 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/391723a7-7e4d-4727-af52-abe87e1ab8ff-openshift-service-ca\") pod \"maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x\" (UID: \"391723a7-7e4d-4727-af52-abe87e1ab8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" Oct 30 13:10:58.649645 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.649606 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/391723a7-7e4d-4727-af52-abe87e1ab8ff-metrics-certs\") pod \"maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x\" (UID: \"391723a7-7e4d-4727-af52-abe87e1ab8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" Oct 30 13:10:58.649867 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.649700 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/391723a7-7e4d-4727-af52-abe87e1ab8ff-openshift-service-ca\") pod \"maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x\" (UID: \"391723a7-7e4d-4727-af52-abe87e1ab8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" Oct 30 13:10:58.649867 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.649829 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk6m7\" (UniqueName: \"kubernetes.io/projected/391723a7-7e4d-4727-af52-abe87e1ab8ff-kube-api-access-qk6m7\") pod \"maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x\" (UID: \"391723a7-7e4d-4727-af52-abe87e1ab8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" Oct 30 13:10:58.650028 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.649868 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/391723a7-7e4d-4727-af52-abe87e1ab8ff-maas-api-tls\") pod \"maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x\" (UID: \"391723a7-7e4d-4727-af52-abe87e1ab8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" Oct 30 13:10:58.650603 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.650571 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/391723a7-7e4d-4727-af52-abe87e1ab8ff-openshift-service-ca\") pod \"maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x\" (UID: \"391723a7-7e4d-4727-af52-abe87e1ab8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" Oct 30 13:10:58.652386 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.652361 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/391723a7-7e4d-4727-af52-abe87e1ab8ff-metrics-certs\") pod \"maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x\" (UID: \"391723a7-7e4d-4727-af52-abe87e1ab8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" Oct 30 13:10:58.652814 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.652794 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/391723a7-7e4d-4727-af52-abe87e1ab8ff-maas-api-tls\") pod \"maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x\" (UID: \"391723a7-7e4d-4727-af52-abe87e1ab8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" Oct 30 13:10:58.736976 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.733829 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk6m7\" (UniqueName: \"kubernetes.io/projected/391723a7-7e4d-4727-af52-abe87e1ab8ff-kube-api-access-qk6m7\") pod \"maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x\" (UID: \"391723a7-7e4d-4727-af52-abe87e1ab8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" Oct 30 13:10:58.770962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:58.770252 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" Oct 30 13:10:59.210717 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:59.210679 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x"] Oct 30 13:10:59.211166 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:10:59.211142 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391723a7_7e4d_4727_af52_abe87e1ab8ff.slice/crio-0e103c1588543c6742dff3314a9d711552366e7545c9b6da182d5d2e92a9cd96 WatchSource:0}: Error finding container 0e103c1588543c6742dff3314a9d711552366e7545c9b6da182d5d2e92a9cd96: Status 404 returned error can't find the container with id 0e103c1588543c6742dff3314a9d711552366e7545c9b6da182d5d2e92a9cd96 Oct 30 13:10:59.659502 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:10:59.659466 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" event={"ID":"391723a7-7e4d-4727-af52-abe87e1ab8ff","Type":"ContainerStarted","Data":"0e103c1588543c6742dff3314a9d711552366e7545c9b6da182d5d2e92a9cd96"} Oct 30 13:10:59.708953 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:10:59.708901 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:11:00.668633 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:00.668595 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" event={"ID":"391723a7-7e4d-4727-af52-abe87e1ab8ff","Type":"ContainerStarted","Data":"69c5b8eb8be087810af9de827569493ae7d012a2d4b2ecf0cf425de63ee9beb7"} Oct 30 13:11:00.669189 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:00.668815 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" Oct 30 13:11:00.747971 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:00.747075 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" podStartSLOduration=2.349808863 podStartE2EDuration="2.747052278s" totalImagesPullingTime="397.243415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:00.711571253 +0000 UTC m=+1154.534799534" watchObservedRunningTime="2026-10-30 13:11:00.747052278 +0000 UTC m=+1154.570280567" Oct 30 13:11:01.161503 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.161470 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:11:01.310121 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.310078 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8daeac71-175e-4684-84df-b27b0d52e733-istio-token\" (UniqueName: \"kubernetes.io/projected/8daeac71-175e-4684-84df-b27b0d52e733-istio-token\") pod \"8daeac71-175e-4684-84df-b27b0d52e733\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " Oct 30 13:11:01.310314 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.310189 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8daeac71-175e-4684-84df-b27b0d52e733-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8daeac71-175e-4684-84df-b27b0d52e733-istio-podinfo\") pod \"8daeac71-175e-4684-84df-b27b0d52e733\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " Oct 30 13:11:01.310314 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.310264 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-istio-envoy\") pod \"8daeac71-175e-4684-84df-b27b0d52e733\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " Oct 30 13:11:01.310446 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.310328 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8daeac71-175e-4684-84df-b27b0d52e733-kube-api-access-tpx2g\" (UniqueName: \"kubernetes.io/projected/8daeac71-175e-4684-84df-b27b0d52e733-kube-api-access-tpx2g\") pod \"8daeac71-175e-4684-84df-b27b0d52e733\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " Oct 30 13:11:01.310446 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.310366 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-workload-socket\") pod \"8daeac71-175e-4684-84df-b27b0d52e733\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " Oct 30 13:11:01.310549 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.310494 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-istio-data\") pod \"8daeac71-175e-4684-84df-b27b0d52e733\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " Oct 30 13:11:01.310911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.310704 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-workload-socket" pod "8daeac71-175e-4684-84df-b27b0d52e733" (UID: "8daeac71-175e-4684-84df-b27b0d52e733"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.310911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.310746 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-credential-socket\") pod \"8daeac71-175e-4684-84df-b27b0d52e733\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " Oct 30 13:11:01.310911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.310850 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-workload-certs\") pod \"8daeac71-175e-4684-84df-b27b0d52e733\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " Oct 30 13:11:01.311352 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.310843 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-istio-data" pod "8daeac71-175e-4684-84df-b27b0d52e733" (UID: "8daeac71-175e-4684-84df-b27b0d52e733"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.311429 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.311020 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-credential-socket" pod "8daeac71-175e-4684-84df-b27b0d52e733" (UID: "8daeac71-175e-4684-84df-b27b0d52e733"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.311429 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.311256 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-workload-certs" pod "8daeac71-175e-4684-84df-b27b0d52e733" (UID: "8daeac71-175e-4684-84df-b27b0d52e733"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.312102 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.312076 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8daeac71-175e-4684-84df-b27b0d52e733-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8daeac71-175e-4684-84df-b27b0d52e733-istiod-ca-cert\") pod \"8daeac71-175e-4684-84df-b27b0d52e733\" (UID: \"8daeac71-175e-4684-84df-b27b0d52e733\") " Oct 30 13:11:01.312957 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.312917 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.313209 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.313191 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.313285 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.313214 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.313285 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.313227 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.313285 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.313110 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8daeac71-175e-4684-84df-b27b0d52e733-istio-token" pod "8daeac71-175e-4684-84df-b27b0d52e733" (UID: "8daeac71-175e-4684-84df-b27b0d52e733"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:01.313424 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.313291 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8daeac71-175e-4684-84df-b27b0d52e733-istiod-ca-cert" pod "8daeac71-175e-4684-84df-b27b0d52e733" (UID: "8daeac71-175e-4684-84df-b27b0d52e733"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:01.313480 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.313464 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-istio-envoy" pod "8daeac71-175e-4684-84df-b27b0d52e733" (UID: "8daeac71-175e-4684-84df-b27b0d52e733"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.314122 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.314088 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8daeac71-175e-4684-84df-b27b0d52e733-istio-podinfo" pod "8daeac71-175e-4684-84df-b27b0d52e733" (UID: "8daeac71-175e-4684-84df-b27b0d52e733"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:01.315167 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.315132 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8daeac71-175e-4684-84df-b27b0d52e733-kube-api-access-tpx2g" pod "8daeac71-175e-4684-84df-b27b0d52e733" (UID: "8daeac71-175e-4684-84df-b27b0d52e733"). InnerVolumeSpecName "kube-api-access-tpx2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:01.414670 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.414632 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8daeac71-175e-4684-84df-b27b0d52e733-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.414803 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.414674 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8daeac71-175e-4684-84df-b27b0d52e733-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.414803 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.414690 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpx2g\" (UniqueName: \"kubernetes.io/projected/8daeac71-175e-4684-84df-b27b0d52e733-kube-api-access-tpx2g\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.414803 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.414705 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8daeac71-175e-4684-84df-b27b0d52e733-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.414803 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.414717 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8daeac71-175e-4684-84df-b27b0d52e733-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.595195 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.595169 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:11:01.690854 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.690735 4503 generic.go:356] "Generic (PLEG): container finished" podID="8daeac71-175e-4684-84df-b27b0d52e733" containerID="2a3dbc6b593fa6b5f6af8b9510907704a87be349f9a8c5fd2219811b1ab2faf0" exitCode=0 Oct 30 13:11:01.691514 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.691079 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" event={"ID":"8daeac71-175e-4684-84df-b27b0d52e733","Type":"ContainerDied","Data":"2a3dbc6b593fa6b5f6af8b9510907704a87be349f9a8c5fd2219811b1ab2faf0"} Oct 30 13:11:01.691514 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.691161 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" event={"ID":"8daeac71-175e-4684-84df-b27b0d52e733","Type":"ContainerDied","Data":"cd2c183b0f6e0a5038eadaac2be3026ecd4c85b5a83d4bc79ed153d9fbab1e6e"} Oct 30 13:11:01.691514 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.691210 4503 scope.go:122] "RemoveContainer" containerID="2a3dbc6b593fa6b5f6af8b9510907704a87be349f9a8c5fd2219811b1ab2faf0" Oct 30 13:11:01.692091 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.692068 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx" Oct 30 13:11:01.696126 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.696096 4503 generic.go:356] "Generic (PLEG): container finished" podID="dc8ab9ac-098f-4b41-9f0d-51422627d481" containerID="0f240e34b87ea0291df9fef93f44dfc807ce3b33cbd42f0f63857773ce414673" exitCode=0 Oct 30 13:11:01.696262 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.696215 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" event={"ID":"dc8ab9ac-098f-4b41-9f0d-51422627d481","Type":"ContainerDied","Data":"0f240e34b87ea0291df9fef93f44dfc807ce3b33cbd42f0f63857773ce414673"} Oct 30 13:11:01.698173 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.698152 4503 generic.go:356] "Generic (PLEG): container finished" podID="d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" containerID="994775eeb5e92033429ace2d3d4fdf0d07b3edfa3e9be5b169f386c682316634" exitCode=0 Oct 30 13:11:01.698596 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.698212 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" event={"ID":"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf","Type":"ContainerDied","Data":"994775eeb5e92033429ace2d3d4fdf0d07b3edfa3e9be5b169f386c682316634"} Oct 30 13:11:01.698596 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.698402 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" event={"ID":"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf","Type":"ContainerDied","Data":"ee595bd35c57acf192d59cf42d4dea0dee8c7b5b799a16e8d05945d6dd993eac"} Oct 30 13:11:01.698596 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.698272 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd" Oct 30 13:11:01.700880 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.700858 4503 generic.go:356] "Generic (PLEG): container finished" podID="bec8b24c-a530-4514-9f0b-5a11af7776ce" containerID="7398c73c609808de1e473d0405e9e5c221e59ad2e2c2cd09544fd166387362e3" exitCode=0 Oct 30 13:11:01.701010 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.700981 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" event={"ID":"bec8b24c-a530-4514-9f0b-5a11af7776ce","Type":"ContainerDied","Data":"7398c73c609808de1e473d0405e9e5c221e59ad2e2c2cd09544fd166387362e3"} Oct 30 13:11:01.709880 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.709765 4503 scope.go:122] "RemoveContainer" containerID="2a3dbc6b593fa6b5f6af8b9510907704a87be349f9a8c5fd2219811b1ab2faf0" Oct 30 13:11:01.710458 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:11:01.710287 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a3dbc6b593fa6b5f6af8b9510907704a87be349f9a8c5fd2219811b1ab2faf0\": container with ID starting with 2a3dbc6b593fa6b5f6af8b9510907704a87be349f9a8c5fd2219811b1ab2faf0 not found: ID does not exist" containerID="2a3dbc6b593fa6b5f6af8b9510907704a87be349f9a8c5fd2219811b1ab2faf0" Oct 30 13:11:01.710458 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.710324 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a3dbc6b593fa6b5f6af8b9510907704a87be349f9a8c5fd2219811b1ab2faf0"} err="failed to get container status \"2a3dbc6b593fa6b5f6af8b9510907704a87be349f9a8c5fd2219811b1ab2faf0\": rpc error: code = NotFound desc = could not find container \"2a3dbc6b593fa6b5f6af8b9510907704a87be349f9a8c5fd2219811b1ab2faf0\": container with ID starting with 2a3dbc6b593fa6b5f6af8b9510907704a87be349f9a8c5fd2219811b1ab2faf0 not found: ID does not exist" Oct 30 13:11:01.710458 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.710349 4503 scope.go:122] "RemoveContainer" containerID="994775eeb5e92033429ace2d3d4fdf0d07b3edfa3e9be5b169f386c682316634" Oct 30 13:11:01.723118 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.721777 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-kube-api-access-8p5qb\" (UniqueName: \"kubernetes.io/projected/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-kube-api-access-8p5qb\") pod \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " Oct 30 13:11:01.723118 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.721867 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-data\") pod \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " Oct 30 13:11:01.723118 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.722020 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-workload-certs\") pod \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " Oct 30 13:11:01.723118 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.722172 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-envoy\") pod \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " Oct 30 13:11:01.723118 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.722254 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-workload-socket\") pod \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " Oct 30 13:11:01.723118 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.722320 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-podinfo\") pod \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " Oct 30 13:11:01.723118 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.722381 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-credential-socket\") pod \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " Oct 30 13:11:01.723118 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.722453 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-token\" (UniqueName: \"kubernetes.io/projected/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-token\") pod \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " Oct 30 13:11:01.723118 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.722488 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istiod-ca-cert\") pod \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\" (UID: \"d3a6bc11-e10e-42aa-a182-38c19cbc9dcf\") " Oct 30 13:11:01.724100 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.723231 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-credential-socket" pod "d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" (UID: "d3a6bc11-e10e-42aa-a182-38c19cbc9dcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.724100 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.723446 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-workload-socket" pod "d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" (UID: "d3a6bc11-e10e-42aa-a182-38c19cbc9dcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.724100 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.723921 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-data" pod "d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" (UID: "d3a6bc11-e10e-42aa-a182-38c19cbc9dcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.724100 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.724068 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istiod-ca-cert" pod "d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" (UID: "d3a6bc11-e10e-42aa-a182-38c19cbc9dcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:01.724372 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.724217 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.724372 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.724242 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.724372 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.724258 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.724372 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.724321 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-workload-certs" pod "d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" (UID: "d3a6bc11-e10e-42aa-a182-38c19cbc9dcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.726205 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.725900 4503 scope.go:122] "RemoveContainer" containerID="994775eeb5e92033429ace2d3d4fdf0d07b3edfa3e9be5b169f386c682316634" Oct 30 13:11:01.726337 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.726288 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-podinfo" pod "d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" (UID: "d3a6bc11-e10e-42aa-a182-38c19cbc9dcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:01.727001 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:11:01.726884 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"994775eeb5e92033429ace2d3d4fdf0d07b3edfa3e9be5b169f386c682316634\": container with ID starting with 994775eeb5e92033429ace2d3d4fdf0d07b3edfa3e9be5b169f386c682316634 not found: ID does not exist" containerID="994775eeb5e92033429ace2d3d4fdf0d07b3edfa3e9be5b169f386c682316634" Oct 30 13:11:01.727001 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.726918 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"994775eeb5e92033429ace2d3d4fdf0d07b3edfa3e9be5b169f386c682316634"} err="failed to get container status \"994775eeb5e92033429ace2d3d4fdf0d07b3edfa3e9be5b169f386c682316634\": rpc error: code = NotFound desc = could not find container \"994775eeb5e92033429ace2d3d4fdf0d07b3edfa3e9be5b169f386c682316634\": container with ID starting with 994775eeb5e92033429ace2d3d4fdf0d07b3edfa3e9be5b169f386c682316634 not found: ID does not exist" Oct 30 13:11:01.727670 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.727641 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-token" pod "d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" (UID: "d3a6bc11-e10e-42aa-a182-38c19cbc9dcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:01.728073 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.728033 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-envoy" pod "d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" (UID: "d3a6bc11-e10e-42aa-a182-38c19cbc9dcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.736046 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.735467 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-kube-api-access-8p5qb" pod "d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" (UID: "d3a6bc11-e10e-42aa-a182-38c19cbc9dcf"). InnerVolumeSpecName "kube-api-access-8p5qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:01.760147 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.754267 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:11:01.760147 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.756782 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx"] Oct 30 13:11:01.779019 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.762270 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zxmjx"] Oct 30 13:11:01.820696 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.820396 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:11:01.825857 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.825188 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-workload-socket\") pod \"dc8ab9ac-098f-4b41-9f0d-51422627d481\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " Oct 30 13:11:01.825857 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.825228 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc8ab9ac-098f-4b41-9f0d-51422627d481-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8ab9ac-098f-4b41-9f0d-51422627d481-istiod-ca-cert\") pod \"dc8ab9ac-098f-4b41-9f0d-51422627d481\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " Oct 30 13:11:01.825857 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.825336 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-podinfo\") pod \"dc8ab9ac-098f-4b41-9f0d-51422627d481\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " Oct 30 13:11:01.825857 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.825365 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-envoy\") pod \"dc8ab9ac-098f-4b41-9f0d-51422627d481\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " Oct 30 13:11:01.825857 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.825428 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-workload-certs\") pod \"dc8ab9ac-098f-4b41-9f0d-51422627d481\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " Oct 30 13:11:01.825857 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.825452 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-data\") pod \"dc8ab9ac-098f-4b41-9f0d-51422627d481\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " Oct 30 13:11:01.825857 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.825478 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc8ab9ac-098f-4b41-9f0d-51422627d481-kube-api-access-285kq\" (UniqueName: \"kubernetes.io/projected/dc8ab9ac-098f-4b41-9f0d-51422627d481-kube-api-access-285kq\") pod \"dc8ab9ac-098f-4b41-9f0d-51422627d481\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " Oct 30 13:11:01.825857 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.825509 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-workload-socket" pod "dc8ab9ac-098f-4b41-9f0d-51422627d481" (UID: "dc8ab9ac-098f-4b41-9f0d-51422627d481"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.825857 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.825672 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-credential-socket\") pod \"dc8ab9ac-098f-4b41-9f0d-51422627d481\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " Oct 30 13:11:01.825857 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.825758 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-token\" (UniqueName: \"kubernetes.io/projected/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-token\") pod \"dc8ab9ac-098f-4b41-9f0d-51422627d481\" (UID: \"dc8ab9ac-098f-4b41-9f0d-51422627d481\") " Oct 30 13:11:01.825857 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.825782 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-workload-certs" pod "dc8ab9ac-098f-4b41-9f0d-51422627d481" (UID: "dc8ab9ac-098f-4b41-9f0d-51422627d481"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.825857 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.825791 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc8ab9ac-098f-4b41-9f0d-51422627d481-istiod-ca-cert" pod "dc8ab9ac-098f-4b41-9f0d-51422627d481" (UID: "dc8ab9ac-098f-4b41-9f0d-51422627d481"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:01.826555 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.826073 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-credential-socket" pod "dc8ab9ac-098f-4b41-9f0d-51422627d481" (UID: "dc8ab9ac-098f-4b41-9f0d-51422627d481"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.826555 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.826145 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-data" pod "dc8ab9ac-098f-4b41-9f0d-51422627d481" (UID: "dc8ab9ac-098f-4b41-9f0d-51422627d481"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.826899 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.826872 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.827023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.826903 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.827023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.826919 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.827023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.826934 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.827023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.826963 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.827023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.826975 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.827023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.826987 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.827023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.827000 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8ab9ac-098f-4b41-9f0d-51422627d481-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.827023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.827014 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.827023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.827027 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p5qb\" (UniqueName: \"kubernetes.io/projected/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf-kube-api-access-8p5qb\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.827470 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.827039 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.829011 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.828964 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8ab9ac-098f-4b41-9f0d-51422627d481-kube-api-access-285kq" pod "dc8ab9ac-098f-4b41-9f0d-51422627d481" (UID: "dc8ab9ac-098f-4b41-9f0d-51422627d481"). InnerVolumeSpecName "kube-api-access-285kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:01.829126 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.829066 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-envoy" pod "dc8ab9ac-098f-4b41-9f0d-51422627d481" (UID: "dc8ab9ac-098f-4b41-9f0d-51422627d481"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.829364 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.829342 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-podinfo" pod "dc8ab9ac-098f-4b41-9f0d-51422627d481" (UID: "dc8ab9ac-098f-4b41-9f0d-51422627d481"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:01.830466 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.830445 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-token" pod "dc8ab9ac-098f-4b41-9f0d-51422627d481" (UID: "dc8ab9ac-098f-4b41-9f0d-51422627d481"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:01.928713 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.928666 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-envoy\") pod \"bec8b24c-a530-4514-9f0b-5a11af7776ce\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " Oct 30 13:11:01.928896 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.928781 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-workload-certs\") pod \"bec8b24c-a530-4514-9f0b-5a11af7776ce\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " Oct 30 13:11:01.928896 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.928825 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-podinfo\") pod \"bec8b24c-a530-4514-9f0b-5a11af7776ce\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " Oct 30 13:11:01.928896 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.928888 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-token\" (UniqueName: \"kubernetes.io/projected/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-token\") pod \"bec8b24c-a530-4514-9f0b-5a11af7776ce\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " Oct 30 13:11:01.929137 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.928931 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-credential-socket\") pod \"bec8b24c-a530-4514-9f0b-5a11af7776ce\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " Oct 30 13:11:01.929137 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.929089 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bec8b24c-a530-4514-9f0b-5a11af7776ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec8b24c-a530-4514-9f0b-5a11af7776ce-istiod-ca-cert\") pod \"bec8b24c-a530-4514-9f0b-5a11af7776ce\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " Oct 30 13:11:01.929256 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.929139 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-data\") pod \"bec8b24c-a530-4514-9f0b-5a11af7776ce\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " Oct 30 13:11:01.929256 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.929172 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-workload-socket\") pod \"bec8b24c-a530-4514-9f0b-5a11af7776ce\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " Oct 30 13:11:01.929363 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.929251 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec8b24c-a530-4514-9f0b-5a11af7776ce-kube-api-access-v5mfw\" (UniqueName: \"kubernetes.io/projected/bec8b24c-a530-4514-9f0b-5a11af7776ce-kube-api-access-v5mfw\") pod \"bec8b24c-a530-4514-9f0b-5a11af7776ce\" (UID: \"bec8b24c-a530-4514-9f0b-5a11af7776ce\") " Oct 30 13:11:01.929632 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.929607 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-data" pod "bec8b24c-a530-4514-9f0b-5a11af7776ce" (UID: "bec8b24c-a530-4514-9f0b-5a11af7776ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.929722 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.929630 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bec8b24c-a530-4514-9f0b-5a11af7776ce-istiod-ca-cert" pod "bec8b24c-a530-4514-9f0b-5a11af7776ce" (UID: "bec8b24c-a530-4514-9f0b-5a11af7776ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:01.929858 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.929839 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-credential-socket" pod "bec8b24c-a530-4514-9f0b-5a11af7776ce" (UID: "bec8b24c-a530-4514-9f0b-5a11af7776ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.929924 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.929889 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-workload-socket" pod "bec8b24c-a530-4514-9f0b-5a11af7776ce" (UID: "bec8b24c-a530-4514-9f0b-5a11af7776ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.930216 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.930199 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.930216 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.930206 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-workload-certs" pod "bec8b24c-a530-4514-9f0b-5a11af7776ce" (UID: "bec8b24c-a530-4514-9f0b-5a11af7776ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.930357 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.930221 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec8b24c-a530-4514-9f0b-5a11af7776ce-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.930357 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.930235 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.930357 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.930247 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.930357 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.930259 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.930357 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.930274 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.930357 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.930289 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-285kq\" (UniqueName: \"kubernetes.io/projected/dc8ab9ac-098f-4b41-9f0d-51422627d481-kube-api-access-285kq\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.930357 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.930301 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8ab9ac-098f-4b41-9f0d-51422627d481-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:01.931548 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.931513 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-envoy" pod "bec8b24c-a530-4514-9f0b-5a11af7776ce" (UID: "bec8b24c-a530-4514-9f0b-5a11af7776ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:01.932394 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.932363 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-podinfo" pod "bec8b24c-a530-4514-9f0b-5a11af7776ce" (UID: "bec8b24c-a530-4514-9f0b-5a11af7776ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:01.932507 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.932411 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec8b24c-a530-4514-9f0b-5a11af7776ce-kube-api-access-v5mfw" pod "bec8b24c-a530-4514-9f0b-5a11af7776ce" (UID: "bec8b24c-a530-4514-9f0b-5a11af7776ce"). InnerVolumeSpecName "kube-api-access-v5mfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:01.932686 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:01.932670 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-token" pod "bec8b24c-a530-4514-9f0b-5a11af7776ce" (UID: "bec8b24c-a530-4514-9f0b-5a11af7776ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:02.031553 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.031516 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5mfw\" (UniqueName: \"kubernetes.io/projected/bec8b24c-a530-4514-9f0b-5a11af7776ce-kube-api-access-v5mfw\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:02.031553 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.031554 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:02.031768 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.031565 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec8b24c-a530-4514-9f0b-5a11af7776ce-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:02.031768 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.031592 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:02.031768 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.031601 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec8b24c-a530-4514-9f0b-5a11af7776ce-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:02.050539 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.050509 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd"] Oct 30 13:11:02.054363 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.054338 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qvxgd"] Oct 30 13:11:02.366857 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.366773 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8daeac71-175e-4684-84df-b27b0d52e733" path="/var/lib/kubelet/pods/8daeac71-175e-4684-84df-b27b0d52e733/volumes" Oct 30 13:11:02.367603 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.367584 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" path="/var/lib/kubelet/pods/d3a6bc11-e10e-42aa-a182-38c19cbc9dcf/volumes" Oct 30 13:11:02.712538 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.712488 4503 generic.go:356] "Generic (PLEG): container finished" podID="0549ac29-5939-4f6c-aced-091b158f4afd" containerID="f65cd9f96200812784e430e43dc23a011dd406b61d3d2f98c4a952aa3149b619" exitCode=0 Oct 30 13:11:02.712856 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.712564 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" event={"ID":"0549ac29-5939-4f6c-aced-091b158f4afd","Type":"ContainerDied","Data":"f65cd9f96200812784e430e43dc23a011dd406b61d3d2f98c4a952aa3149b619"} Oct 30 13:11:02.712856 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.712682 4503 scope.go:122] "RemoveContainer" containerID="bdd197d00bc095e8f4d4a0a9a86e5c64e90b853d648b65df3aea523182e18ea8" Oct 30 13:11:02.715522 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.715503 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" Oct 30 13:11:02.715688 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.715499 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6" event={"ID":"dc8ab9ac-098f-4b41-9f0d-51422627d481","Type":"ContainerDied","Data":"e74a4b7f3bc8262c938dbdc93b1be746421cbd72ddf1d9959206c21eea7fa4b9"} Oct 30 13:11:02.718550 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.718519 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" event={"ID":"bec8b24c-a530-4514-9f0b-5a11af7776ce","Type":"ContainerDied","Data":"1a7ea3d3562fa051f8db10a54c1161534bcab611530fed97d06af69bc9005998"} Oct 30 13:11:02.718648 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.718600 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4" Oct 30 13:11:02.733235 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.733214 4503 scope.go:122] "RemoveContainer" containerID="0f240e34b87ea0291df9fef93f44dfc807ce3b33cbd42f0f63857773ce414673" Oct 30 13:11:02.751004 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.748364 4503 scope.go:122] "RemoveContainer" containerID="7398c73c609808de1e473d0405e9e5c221e59ad2e2c2cd09544fd166387362e3" Oct 30 13:11:02.778278 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.778240 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6"] Oct 30 13:11:02.951600 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.951500 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mbnm6"] Oct 30 13:11:02.957748 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.957710 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4"] Oct 30 13:11:02.965225 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:02.965081 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nm7s4"] Oct 30 13:11:03.071289 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.071261 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:11:03.154813 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.154773 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-istio-envoy\") pod \"0549ac29-5939-4f6c-aced-091b158f4afd\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " Oct 30 13:11:03.155037 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.154911 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0549ac29-5939-4f6c-aced-091b158f4afd-istio-token\" (UniqueName: \"kubernetes.io/projected/0549ac29-5939-4f6c-aced-091b158f4afd-istio-token\") pod \"0549ac29-5939-4f6c-aced-091b158f4afd\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " Oct 30 13:11:03.155037 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.155017 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-istio-data\") pod \"0549ac29-5939-4f6c-aced-091b158f4afd\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " Oct 30 13:11:03.155165 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.155076 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-workload-socket\") pod \"0549ac29-5939-4f6c-aced-091b158f4afd\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " Oct 30 13:11:03.155165 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.155127 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0549ac29-5939-4f6c-aced-091b158f4afd-kube-api-access-p96qc\" (UniqueName: \"kubernetes.io/projected/0549ac29-5939-4f6c-aced-091b158f4afd-kube-api-access-p96qc\") pod \"0549ac29-5939-4f6c-aced-091b158f4afd\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " Oct 30 13:11:03.155275 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.155166 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-credential-socket\") pod \"0549ac29-5939-4f6c-aced-091b158f4afd\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " Oct 30 13:11:03.155651 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.155382 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-workload-socket" pod "0549ac29-5939-4f6c-aced-091b158f4afd" (UID: "0549ac29-5939-4f6c-aced-091b158f4afd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:03.155651 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.155453 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0549ac29-5939-4f6c-aced-091b158f4afd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0549ac29-5939-4f6c-aced-091b158f4afd-istio-podinfo\") pod \"0549ac29-5939-4f6c-aced-091b158f4afd\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " Oct 30 13:11:03.155651 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.155520 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-workload-certs\") pod \"0549ac29-5939-4f6c-aced-091b158f4afd\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " Oct 30 13:11:03.155651 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.155563 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0549ac29-5939-4f6c-aced-091b158f4afd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0549ac29-5939-4f6c-aced-091b158f4afd-istiod-ca-cert\") pod \"0549ac29-5939-4f6c-aced-091b158f4afd\" (UID: \"0549ac29-5939-4f6c-aced-091b158f4afd\") " Oct 30 13:11:03.155651 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.155647 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-credential-socket" pod "0549ac29-5939-4f6c-aced-091b158f4afd" (UID: "0549ac29-5939-4f6c-aced-091b158f4afd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:03.156380 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.156359 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:03.156489 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.156384 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:03.156489 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.156363 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-workload-certs" pod "0549ac29-5939-4f6c-aced-091b158f4afd" (UID: "0549ac29-5939-4f6c-aced-091b158f4afd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:03.156663 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.156636 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0549ac29-5939-4f6c-aced-091b158f4afd-istiod-ca-cert" pod "0549ac29-5939-4f6c-aced-091b158f4afd" (UID: "0549ac29-5939-4f6c-aced-091b158f4afd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:03.157604 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.157576 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-istio-envoy" pod "0549ac29-5939-4f6c-aced-091b158f4afd" (UID: "0549ac29-5939-4f6c-aced-091b158f4afd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:03.158030 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.158005 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0549ac29-5939-4f6c-aced-091b158f4afd-istio-token" pod "0549ac29-5939-4f6c-aced-091b158f4afd" (UID: "0549ac29-5939-4f6c-aced-091b158f4afd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:03.158250 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.158224 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-istio-data" pod "0549ac29-5939-4f6c-aced-091b158f4afd" (UID: "0549ac29-5939-4f6c-aced-091b158f4afd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:03.158450 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.158431 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0549ac29-5939-4f6c-aced-091b158f4afd-kube-api-access-p96qc" pod "0549ac29-5939-4f6c-aced-091b158f4afd" (UID: "0549ac29-5939-4f6c-aced-091b158f4afd"). InnerVolumeSpecName "kube-api-access-p96qc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:03.158809 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.158793 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0549ac29-5939-4f6c-aced-091b158f4afd-istio-podinfo" pod "0549ac29-5939-4f6c-aced-091b158f4afd" (UID: "0549ac29-5939-4f6c-aced-091b158f4afd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:03.257517 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.257416 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:03.257517 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.257451 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0549ac29-5939-4f6c-aced-091b158f4afd-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:03.257517 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.257462 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:03.257517 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.257471 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p96qc\" (UniqueName: \"kubernetes.io/projected/0549ac29-5939-4f6c-aced-091b158f4afd-kube-api-access-p96qc\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:03.257517 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.257482 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0549ac29-5939-4f6c-aced-091b158f4afd-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:03.257517 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.257491 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0549ac29-5939-4f6c-aced-091b158f4afd-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:03.257517 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.257500 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0549ac29-5939-4f6c-aced-091b158f4afd-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:03.643888 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.643858 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" podUID="803ac7a6-439b-4634-a116-5e16c2595f42" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:11:03.730720 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.730680 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" event={"ID":"0549ac29-5939-4f6c-aced-091b158f4afd","Type":"ContainerDied","Data":"b3bccbea15b4857e762d15f16b00fbb9336ecad05514f3b0dd85e1dd3fd8a583"} Oct 30 13:11:03.731245 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.730756 4503 scope.go:122] "RemoveContainer" containerID="f65cd9f96200812784e430e43dc23a011dd406b61d3d2f98c4a952aa3149b619" Oct 30 13:11:03.731245 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.730697 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2" Oct 30 13:11:03.789977 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.789910 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2"] Oct 30 13:11:03.802214 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:03.802175 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-253f7c37-openshift-default-6b46779fcc-sq8v2"] Oct 30 13:11:04.364545 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:04.364510 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0549ac29-5939-4f6c-aced-091b158f4afd" path="/var/lib/kubelet/pods/0549ac29-5939-4f6c-aced-091b158f4afd/volumes" Oct 30 13:11:04.365246 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:04.365221 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bec8b24c-a530-4514-9f0b-5a11af7776ce" path="/var/lib/kubelet/pods/bec8b24c-a530-4514-9f0b-5a11af7776ce/volumes" Oct 30 13:11:04.365772 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:04.365745 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc8ab9ac-098f-4b41-9f0d-51422627d481" path="/var/lib/kubelet/pods/dc8ab9ac-098f-4b41-9f0d-51422627d481/volumes" Oct 30 13:11:05.431981 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:05.431932 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t"] Oct 30 13:11:05.432625 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:05.432220 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" podUID="52f2532a-6a21-4115-8b67-082b8b9ea83f" containerName="istio-proxy" containerID="cri-o://d431d4727b5ac6eb489f50ff7641f585b67b64ccb7ab7f45ad1fc8e5c0be2a31" gracePeriod=30 Oct 30 13:11:06.707693 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:06.707662 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" Oct 30 13:11:10.753628 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.753590 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:11:10.789958 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.789903 4503 generic.go:356] "Generic (PLEG): container finished" podID="52f2532a-6a21-4115-8b67-082b8b9ea83f" containerID="d431d4727b5ac6eb489f50ff7641f585b67b64ccb7ab7f45ad1fc8e5c0be2a31" exitCode=0 Oct 30 13:11:10.790151 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.790008 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" Oct 30 13:11:10.790151 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.790007 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" event={"ID":"52f2532a-6a21-4115-8b67-082b8b9ea83f","Type":"ContainerDied","Data":"d431d4727b5ac6eb489f50ff7641f585b67b64ccb7ab7f45ad1fc8e5c0be2a31"} Oct 30 13:11:10.790151 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.790124 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t" event={"ID":"52f2532a-6a21-4115-8b67-082b8b9ea83f","Type":"ContainerDied","Data":"c60f52f5479b4a96b615c97ae9895b9e4fc3cadaf1bb7429fd2318a585e16e76"} Oct 30 13:11:10.790325 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.790153 4503 scope.go:122] "RemoveContainer" containerID="d431d4727b5ac6eb489f50ff7641f585b67b64ccb7ab7f45ad1fc8e5c0be2a31" Oct 30 13:11:10.801388 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.801359 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh"] Oct 30 13:11:10.802401 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.802298 4503 cpu_manager.go:414] "containerMap: removing container" podUID="52f2532a-6a21-4115-8b67-082b8b9ea83f" containerName="istio-proxy" Oct 30 13:11:10.802587 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.802559 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f2532a-6a21-4115-8b67-082b8b9ea83f" containerName="istio-proxy" Oct 30 13:11:10.802699 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.802597 4503 cpu_manager.go:414] "containerMap: removing container" podUID="52f2532a-6a21-4115-8b67-082b8b9ea83f" containerName="istio-proxy" Oct 30 13:11:10.802699 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.802605 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f2532a-6a21-4115-8b67-082b8b9ea83f" containerName="istio-proxy" Oct 30 13:11:10.802699 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.802618 4503 cpu_manager.go:414] "containerMap: removing container" podUID="0549ac29-5939-4f6c-aced-091b158f4afd" containerName="istio-proxy" Oct 30 13:11:10.802699 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.802627 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0549ac29-5939-4f6c-aced-091b158f4afd" containerName="istio-proxy" Oct 30 13:11:10.802699 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.802647 4503 cpu_manager.go:414] "containerMap: removing container" podUID="bec8b24c-a530-4514-9f0b-5a11af7776ce" containerName="istio-proxy" Oct 30 13:11:10.802699 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.802654 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec8b24c-a530-4514-9f0b-5a11af7776ce" containerName="istio-proxy" Oct 30 13:11:10.802699 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.802670 4503 cpu_manager.go:414] "containerMap: removing container" podUID="d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" containerName="istio-proxy" Oct 30 13:11:10.802699 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.802677 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" containerName="istio-proxy" Oct 30 13:11:10.803269 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.802743 4503 cpu_manager.go:414] "containerMap: removing container" podUID="dc8ab9ac-098f-4b41-9f0d-51422627d481" containerName="istio-proxy" Oct 30 13:11:10.803269 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.802751 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc8ab9ac-098f-4b41-9f0d-51422627d481" containerName="istio-proxy" Oct 30 13:11:10.803269 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.802770 4503 cpu_manager.go:414] "containerMap: removing container" podUID="8daeac71-175e-4684-84df-b27b0d52e733" containerName="istio-proxy" Oct 30 13:11:10.803269 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.802777 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8daeac71-175e-4684-84df-b27b0d52e733" containerName="istio-proxy" Oct 30 13:11:10.803269 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.802990 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="8daeac71-175e-4684-84df-b27b0d52e733" containerName="istio-proxy" Oct 30 13:11:10.803269 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.803012 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f2532a-6a21-4115-8b67-082b8b9ea83f" containerName="istio-proxy" Oct 30 13:11:10.803269 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.803025 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f2532a-6a21-4115-8b67-082b8b9ea83f" containerName="istio-proxy" Oct 30 13:11:10.803269 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.803033 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="0549ac29-5939-4f6c-aced-091b158f4afd" containerName="istio-proxy" Oct 30 13:11:10.803269 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.803043 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec8b24c-a530-4514-9f0b-5a11af7776ce" containerName="istio-proxy" Oct 30 13:11:10.803269 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.803052 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a6bc11-e10e-42aa-a182-38c19cbc9dcf" containerName="istio-proxy" Oct 30 13:11:10.803269 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.803070 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc8ab9ac-098f-4b41-9f0d-51422627d481" containerName="istio-proxy" Oct 30 13:11:10.804374 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.804280 4503 scope.go:122] "RemoveContainer" containerID="ce2add05a4f83079d26a9215f45be73d03e570c33bb4a5642dad2e0df1252656" Oct 30 13:11:10.806829 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.806807 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:10.821352 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.821319 4503 scope.go:122] "RemoveContainer" containerID="d431d4727b5ac6eb489f50ff7641f585b67b64ccb7ab7f45ad1fc8e5c0be2a31" Oct 30 13:11:10.830555 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:11:10.822347 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d431d4727b5ac6eb489f50ff7641f585b67b64ccb7ab7f45ad1fc8e5c0be2a31\": container with ID starting with d431d4727b5ac6eb489f50ff7641f585b67b64ccb7ab7f45ad1fc8e5c0be2a31 not found: ID does not exist" containerID="d431d4727b5ac6eb489f50ff7641f585b67b64ccb7ab7f45ad1fc8e5c0be2a31" Oct 30 13:11:10.830555 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.822387 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d431d4727b5ac6eb489f50ff7641f585b67b64ccb7ab7f45ad1fc8e5c0be2a31"} err="failed to get container status \"d431d4727b5ac6eb489f50ff7641f585b67b64ccb7ab7f45ad1fc8e5c0be2a31\": rpc error: code = NotFound desc = could not find container \"d431d4727b5ac6eb489f50ff7641f585b67b64ccb7ab7f45ad1fc8e5c0be2a31\": container with ID starting with d431d4727b5ac6eb489f50ff7641f585b67b64ccb7ab7f45ad1fc8e5c0be2a31 not found: ID does not exist" Oct 30 13:11:10.830555 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.822413 4503 scope.go:122] "RemoveContainer" containerID="ce2add05a4f83079d26a9215f45be73d03e570c33bb4a5642dad2e0df1252656" Oct 30 13:11:10.830555 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:11:10.822692 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2add05a4f83079d26a9215f45be73d03e570c33bb4a5642dad2e0df1252656\": container with ID starting with ce2add05a4f83079d26a9215f45be73d03e570c33bb4a5642dad2e0df1252656 not found: ID does not exist" containerID="ce2add05a4f83079d26a9215f45be73d03e570c33bb4a5642dad2e0df1252656" Oct 30 13:11:10.830555 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.822719 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2add05a4f83079d26a9215f45be73d03e570c33bb4a5642dad2e0df1252656"} err="failed to get container status \"ce2add05a4f83079d26a9215f45be73d03e570c33bb4a5642dad2e0df1252656\": rpc error: code = NotFound desc = could not find container \"ce2add05a4f83079d26a9215f45be73d03e570c33bb4a5642dad2e0df1252656\": container with ID starting with ce2add05a4f83079d26a9215f45be73d03e570c33bb4a5642dad2e0df1252656 not found: ID does not exist" Oct 30 13:11:10.833530 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.833478 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-ac0c58\"/\"kube-root-ca.crt\"" Oct 30 13:11:10.833666 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.833528 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-ac0c58\"/\"openshift-service-ca.crt\"" Oct 30 13:11:10.833666 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.833661 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ac0c58\"/\"default-dockercfg-k5q4c\"" Oct 30 13:11:10.833787 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.833767 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ac0c58\"/\"distinct-2-w1-ac0c58-kserve-self-signed-certs\"" Oct 30 13:11:10.839347 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.839317 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh"] Oct 30 13:11:10.913265 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.913225 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52f2532a-6a21-4115-8b67-082b8b9ea83f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f2532a-6a21-4115-8b67-082b8b9ea83f-istiod-ca-cert\") pod \"52f2532a-6a21-4115-8b67-082b8b9ea83f\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " Oct 30 13:11:10.913459 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.913318 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-workload-certs\") pod \"52f2532a-6a21-4115-8b67-082b8b9ea83f\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " Oct 30 13:11:10.913459 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.913402 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-podinfo\") pod \"52f2532a-6a21-4115-8b67-082b8b9ea83f\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " Oct 30 13:11:10.913459 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.913433 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f2532a-6a21-4115-8b67-082b8b9ea83f-kube-api-access-nqff6\" (UniqueName: \"kubernetes.io/projected/52f2532a-6a21-4115-8b67-082b8b9ea83f-kube-api-access-nqff6\") pod \"52f2532a-6a21-4115-8b67-082b8b9ea83f\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " Oct 30 13:11:10.913623 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.913493 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-token\" (UniqueName: \"kubernetes.io/projected/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-token\") pod \"52f2532a-6a21-4115-8b67-082b8b9ea83f\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " Oct 30 13:11:10.913623 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.913542 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-credential-socket\") pod \"52f2532a-6a21-4115-8b67-082b8b9ea83f\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " Oct 30 13:11:10.913623 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.913603 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-data\") pod \"52f2532a-6a21-4115-8b67-082b8b9ea83f\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " Oct 30 13:11:10.913744 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.913628 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-envoy\") pod \"52f2532a-6a21-4115-8b67-082b8b9ea83f\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " Oct 30 13:11:10.913744 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.913682 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-workload-socket\") pod \"52f2532a-6a21-4115-8b67-082b8b9ea83f\" (UID: \"52f2532a-6a21-4115-8b67-082b8b9ea83f\") " Oct 30 13:11:10.913744 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.913685 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-workload-certs" pod "52f2532a-6a21-4115-8b67-082b8b9ea83f" (UID: "52f2532a-6a21-4115-8b67-082b8b9ea83f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:10.913744 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.913720 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f2532a-6a21-4115-8b67-082b8b9ea83f-istiod-ca-cert" pod "52f2532a-6a21-4115-8b67-082b8b9ea83f" (UID: "52f2532a-6a21-4115-8b67-082b8b9ea83f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:10.913889 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.913852 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-credential-socket" pod "52f2532a-6a21-4115-8b67-082b8b9ea83f" (UID: "52f2532a-6a21-4115-8b67-082b8b9ea83f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:10.913927 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.913897 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-data" pod "52f2532a-6a21-4115-8b67-082b8b9ea83f" (UID: "52f2532a-6a21-4115-8b67-082b8b9ea83f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:10.914092 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.914069 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-dshm\") pod \"distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:10.914198 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.914179 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-home\") pod \"distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:10.914198 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.914177 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-workload-socket" pod "52f2532a-6a21-4115-8b67-082b8b9ea83f" (UID: "52f2532a-6a21-4115-8b67-082b8b9ea83f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:10.914349 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.914235 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-tmp-dir\") pod \"distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:10.914349 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.914260 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9361bd06-84ab-404c-8893-e61490265ce2-tls-certs\") pod \"distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:10.914499 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.914467 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-model-cache\") pod \"distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:10.914568 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.914547 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8brd\" (UniqueName: \"kubernetes.io/projected/9361bd06-84ab-404c-8893-e61490265ce2-kube-api-access-r8brd\") pod \"distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:10.914752 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.914735 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:10.914824 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.914754 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f2532a-6a21-4115-8b67-082b8b9ea83f-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:10.914824 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.914775 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:10.914824 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.914783 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:10.914824 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.914792 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:10.916169 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.916140 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-podinfo" pod "52f2532a-6a21-4115-8b67-082b8b9ea83f" (UID: "52f2532a-6a21-4115-8b67-082b8b9ea83f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:10.916169 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.916150 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-token" pod "52f2532a-6a21-4115-8b67-082b8b9ea83f" (UID: "52f2532a-6a21-4115-8b67-082b8b9ea83f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:10.916347 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.916316 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-envoy" pod "52f2532a-6a21-4115-8b67-082b8b9ea83f" (UID: "52f2532a-6a21-4115-8b67-082b8b9ea83f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:10.916401 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:10.916366 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f2532a-6a21-4115-8b67-082b8b9ea83f-kube-api-access-nqff6" pod "52f2532a-6a21-4115-8b67-082b8b9ea83f" (UID: "52f2532a-6a21-4115-8b67-082b8b9ea83f"). InnerVolumeSpecName "kube-api-access-nqff6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:11.017205 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.016287 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-dshm\") pod \"distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:11.017205 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.016378 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-home\") pod \"distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:11.017205 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.016417 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-tmp-dir\") pod \"distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:11.017205 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.016448 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9361bd06-84ab-404c-8893-e61490265ce2-tls-certs\") pod \"distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:11.017205 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.016596 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-model-cache\") pod \"distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:11.017205 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.016665 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8brd\" (UniqueName: \"kubernetes.io/projected/9361bd06-84ab-404c-8893-e61490265ce2-kube-api-access-r8brd\") pod \"distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:11.017205 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.016794 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:11.017205 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.016810 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqff6\" (UniqueName: \"kubernetes.io/projected/52f2532a-6a21-4115-8b67-082b8b9ea83f-kube-api-access-nqff6\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:11.017205 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.016826 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:11.017205 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.016840 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f2532a-6a21-4115-8b67-082b8b9ea83f-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:11.023227 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.022469 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-tmp-dir\") pod \"distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:11.023227 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.022781 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-home\") pod \"distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:11.033971 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.032343 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-model-cache\") pod \"distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:11.033971 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.032388 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-dshm\") pod \"distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:11.044968 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.044160 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9361bd06-84ab-404c-8893-e61490265ce2-tls-certs\") pod \"distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:11.053005 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.048846 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8brd\" (UniqueName: \"kubernetes.io/projected/9361bd06-84ab-404c-8893-e61490265ce2-kube-api-access-r8brd\") pod \"distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:11.140193 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.140156 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:11.375239 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.375065 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t"] Oct 30 13:11:11.388122 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.387350 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-fml6t"] Oct 30 13:11:11.650479 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.649648 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh"] Oct 30 13:11:11.650479 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:11:11.650219 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9361bd06_84ab_404c_8893_e61490265ce2.slice/crio-c9b775575f1a262c12f9642ec077f0bd186899e2ab76e646d11fdbd8b0c22b8f WatchSource:0}: Error finding container c9b775575f1a262c12f9642ec077f0bd186899e2ab76e646d11fdbd8b0c22b8f: Status 404 returned error can't find the container with id c9b775575f1a262c12f9642ec077f0bd186899e2ab76e646d11fdbd8b0c22b8f Oct 30 13:11:11.800385 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.800350 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" event={"ID":"9361bd06-84ab-404c-8893-e61490265ce2","Type":"ContainerStarted","Data":"556c17b2d4cc9257106c677da1cae0896e831fa6daaffd51bf7c6ef18d9ede3c"} Oct 30 13:11:11.800873 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.800395 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" event={"ID":"9361bd06-84ab-404c-8893-e61490265ce2","Type":"ContainerStarted","Data":"c9b775575f1a262c12f9642ec077f0bd186899e2ab76e646d11fdbd8b0c22b8f"} Oct 30 13:11:11.843911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:11.843857 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" podStartSLOduration=1.843841452 podStartE2EDuration="1.843841452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:11.842128596 +0000 UTC m=+1165.665356875" watchObservedRunningTime="2026-10-30 13:11:11.843841452 +0000 UTC m=+1165.667069728" Oct 30 13:11:12.147023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:12.146989 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:12.366190 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:12.366155 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52f2532a-6a21-4115-8b67-082b8b9ea83f" path="/var/lib/kubelet/pods/52f2532a-6a21-4115-8b67-082b8b9ea83f/volumes" Oct 30 13:11:12.902404 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:12.902363 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k"] Oct 30 13:11:13.672970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:13.670249 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" podUID="803ac7a6-439b-4634-a116-5e16c2595f42" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:11:15.881427 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:15.881119 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" Oct 30 13:11:15.917524 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:15.917497 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803ac7a6-439b-4634-a116-5e16c2595f42-kube-api-access-vh65k\" (UniqueName: \"kubernetes.io/projected/803ac7a6-439b-4634-a116-5e16c2595f42-kube-api-access-vh65k\") pod \"803ac7a6-439b-4634-a116-5e16c2595f42\" (UID: \"803ac7a6-439b-4634-a116-5e16c2595f42\") " Oct 30 13:11:15.917524 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:15.917532 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/803ac7a6-439b-4634-a116-5e16c2595f42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803ac7a6-439b-4634-a116-5e16c2595f42-plugins-config-volume\") pod \"803ac7a6-439b-4634-a116-5e16c2595f42\" (UID: \"803ac7a6-439b-4634-a116-5e16c2595f42\") " Oct 30 13:11:15.918028 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:15.918004 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803ac7a6-439b-4634-a116-5e16c2595f42-plugins-config-volume" pod "803ac7a6-439b-4634-a116-5e16c2595f42" (UID: "803ac7a6-439b-4634-a116-5e16c2595f42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:15.918191 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:15.918176 4503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803ac7a6-439b-4634-a116-5e16c2595f42-plugins-config-volume\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:15.919807 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:15.919786 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803ac7a6-439b-4634-a116-5e16c2595f42-kube-api-access-vh65k" pod "803ac7a6-439b-4634-a116-5e16c2595f42" (UID: "803ac7a6-439b-4634-a116-5e16c2595f42"). InnerVolumeSpecName "kube-api-access-vh65k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:15.939789 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:15.939751 4503 generic.go:356] "Generic (PLEG): container finished" podID="803ac7a6-439b-4634-a116-5e16c2595f42" containerID="6867ab281b6bf64d6dfb629e3c907371b7390cad875cc8e85e44ca3427e9afac" exitCode=0 Oct 30 13:11:15.939922 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:15.939797 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" event={"ID":"803ac7a6-439b-4634-a116-5e16c2595f42","Type":"ContainerDied","Data":"6867ab281b6bf64d6dfb629e3c907371b7390cad875cc8e85e44ca3427e9afac"} Oct 30 13:11:15.939922 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:15.939820 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" Oct 30 13:11:15.939922 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:15.939841 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv" event={"ID":"803ac7a6-439b-4634-a116-5e16c2595f42","Type":"ContainerDied","Data":"f3c9daae029960745ea1a5f661bb15394bb1c0f1fb66e38bd9db11a5c9620b88"} Oct 30 13:11:15.939922 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:15.939867 4503 scope.go:122] "RemoveContainer" containerID="6867ab281b6bf64d6dfb629e3c907371b7390cad875cc8e85e44ca3427e9afac" Oct 30 13:11:15.996298 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:15.995774 4503 scope.go:122] "RemoveContainer" containerID="6867ab281b6bf64d6dfb629e3c907371b7390cad875cc8e85e44ca3427e9afac" Oct 30 13:11:15.996298 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:11:15.996229 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6867ab281b6bf64d6dfb629e3c907371b7390cad875cc8e85e44ca3427e9afac\": container with ID starting with 6867ab281b6bf64d6dfb629e3c907371b7390cad875cc8e85e44ca3427e9afac not found: ID does not exist" containerID="6867ab281b6bf64d6dfb629e3c907371b7390cad875cc8e85e44ca3427e9afac" Oct 30 13:11:15.996298 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:15.996267 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6867ab281b6bf64d6dfb629e3c907371b7390cad875cc8e85e44ca3427e9afac"} err="failed to get container status \"6867ab281b6bf64d6dfb629e3c907371b7390cad875cc8e85e44ca3427e9afac\": rpc error: code = NotFound desc = could not find container \"6867ab281b6bf64d6dfb629e3c907371b7390cad875cc8e85e44ca3427e9afac\": container with ID starting with 6867ab281b6bf64d6dfb629e3c907371b7390cad875cc8e85e44ca3427e9afac not found: ID does not exist" Oct 30 13:11:16.020068 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:16.020034 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh65k\" (UniqueName: \"kubernetes.io/projected/803ac7a6-439b-4634-a116-5e16c2595f42-kube-api-access-vh65k\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:16.032751 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:16.032720 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv"] Oct 30 13:11:16.038486 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:16.038459 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-42ee9d-67d468bd87-tv8lv"] Oct 30 13:11:16.398679 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:16.398624 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="803ac7a6-439b-4634-a116-5e16c2595f42" path="/var/lib/kubelet/pods/803ac7a6-439b-4634-a116-5e16c2595f42/volumes" Oct 30 13:11:17.903368 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:17.903316 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" podUID="34fa7386-a0cb-4313-86fc-8a82c931598a" containerName="payload-processing" containerID="cri-o://b768bdbc0e3bf9b47cbfbbc68858f4dc7228102f4db076ef8ce6bdd1b925252d" gracePeriod=25 Oct 30 13:11:20.622018 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:20.621931 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" podUID="34fa7386-a0cb-4313-86fc-8a82c931598a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:11:21.146196 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:21.146162 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:21.160408 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:21.160374 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:22.036656 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:22.036625 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:11:23.305029 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:23.304994 4503 patch_prober.go:29] interesting pod/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": read tcp 10.133.0.2:59632->10.133.0.91:15021: read: connection reset by peer" start-of-body= Oct 30 13:11:23.305495 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:23.305057 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" podUID="04fb46b3-a555-41ab-a7be-f57ed8c9deaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": read tcp 10.133.0.2:59632->10.133.0.91:15021: read: connection reset by peer" Oct 30 13:11:24.041970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:24.041914 4503 generic.go:356] "Generic (PLEG): container finished" podID="04fb46b3-a555-41ab-a7be-f57ed8c9deaf" containerID="f024dbffe0f78e624a039c3ffe55080f2d5035d2d6b6b54088feeed6d5d78a96" exitCode=0 Oct 30 13:11:24.042136 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:24.041966 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" event={"ID":"04fb46b3-a555-41ab-a7be-f57ed8c9deaf","Type":"ContainerDied","Data":"f024dbffe0f78e624a039c3ffe55080f2d5035d2d6b6b54088feeed6d5d78a96"} Oct 30 13:11:24.042434 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:24.042420 4503 scope.go:122] "RemoveContainer" containerID="f024dbffe0f78e624a039c3ffe55080f2d5035d2d6b6b54088feeed6d5d78a96" Oct 30 13:11:24.044037 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:24.044006 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:24.044160 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:24.044076 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:24.044160 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:24.044107 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:25.065447 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:25.065404 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" event={"ID":"04fb46b3-a555-41ab-a7be-f57ed8c9deaf","Type":"ContainerStarted","Data":"132f90033fd2e255d137cf3db1a4d4dcdc2f86c05f2e82afa310d0ecbb0379f2"} Oct 30 13:11:25.295955 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:25.295912 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:11:25.296180 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:25.296152 4503 patch_prober.go:29] interesting pod/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 13:11:25.296257 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:25.296210 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" podUID="04fb46b3-a555-41ab-a7be-f57ed8c9deaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 13:11:26.296307 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:26.296274 4503 patch_prober.go:29] interesting pod/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 13:11:26.296694 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:26.296321 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" podUID="04fb46b3-a555-41ab-a7be-f57ed8c9deaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 13:11:27.295715 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:27.295681 4503 patch_prober.go:29] interesting pod/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 13:11:27.295930 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:27.295742 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" podUID="04fb46b3-a555-41ab-a7be-f57ed8c9deaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 13:11:28.321965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:28.308561 4503 patch_prober.go:29] interesting pod/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 13:11:28.321965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:28.308619 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" podUID="04fb46b3-a555-41ab-a7be-f57ed8c9deaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 13:11:29.296540 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:29.296507 4503 patch_prober.go:29] interesting pod/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 13:11:29.296723 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:29.296615 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" podUID="04fb46b3-a555-41ab-a7be-f57ed8c9deaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 13:11:29.707674 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:11:29.707620 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:11:30.296252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:30.296219 4503 patch_prober.go:29] interesting pod/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 13:11:30.296439 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:30.296281 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" podUID="04fb46b3-a555-41ab-a7be-f57ed8c9deaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 13:11:30.620563 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:30.620460 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" podUID="34fa7386-a0cb-4313-86fc-8a82c931598a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:11:31.300816 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:31.300787 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:11:31.301295 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:31.301154 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:11:31.302035 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:31.302014 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:11:37.137771 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:37.137718 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6"] Oct 30 13:11:37.139815 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:37.139751 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" podUID="70f0f890-bb6a-4627-9487-90741d34337c" containerName="istio-proxy" containerID="cri-o://d50382081c56a2f580490f8b8cbd3a4025d957d1c04889923a3389a2cc6b4573" gracePeriod=30 Oct 30 13:11:37.154055 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:37.154023 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj"] Oct 30 13:11:37.154286 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:37.154256 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" podUID="4151e97e-c25a-49af-8268-baa1c493ee4a" containerName="istio-proxy" containerID="cri-o://519b9a429ca3e6daf718e2c698e775b7b40f6e08bbcb820cd1ddbb85934e41ea" gracePeriod=30 Oct 30 13:11:37.156721 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:37.156692 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj"] Oct 30 13:11:37.157044 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:37.157015 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" podUID="f6a2a7df-fe4f-4055-9de6-b1acaae0db54" containerName="istio-proxy" containerID="cri-o://593a2b8b4297111030ace0025a29f43d9f5ccb30aa1e5ee508db2dd9fcc24cd4" gracePeriod=30 Oct 30 13:11:37.167580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:37.167554 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv"] Oct 30 13:11:37.167729 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:37.167709 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" podUID="e0332beb-0b14-493b-9a37-4de43236c083" containerName="istio-proxy" containerID="cri-o://b71c4743a919c38ffd47173db043f957593e20052b590a28054cf60977388916" gracePeriod=30 Oct 30 13:11:38.225513 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:38.225481 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" Oct 30 13:11:38.243280 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:38.243244 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34fa7386-a0cb-4313-86fc-8a82c931598a-kube-api-access-z8rvn\" (UniqueName: \"kubernetes.io/projected/34fa7386-a0cb-4313-86fc-8a82c931598a-kube-api-access-z8rvn\") pod \"34fa7386-a0cb-4313-86fc-8a82c931598a\" (UID: \"34fa7386-a0cb-4313-86fc-8a82c931598a\") " Oct 30 13:11:38.243438 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:38.243402 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34fa7386-a0cb-4313-86fc-8a82c931598a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34fa7386-a0cb-4313-86fc-8a82c931598a-plugins-config-volume\") pod \"34fa7386-a0cb-4313-86fc-8a82c931598a\" (UID: \"34fa7386-a0cb-4313-86fc-8a82c931598a\") " Oct 30 13:11:38.243829 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:38.243804 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34fa7386-a0cb-4313-86fc-8a82c931598a-plugins-config-volume" pod "34fa7386-a0cb-4313-86fc-8a82c931598a" (UID: "34fa7386-a0cb-4313-86fc-8a82c931598a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:38.247467 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:38.244260 4503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34fa7386-a0cb-4313-86fc-8a82c931598a-plugins-config-volume\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:38.251136 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:38.251094 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34fa7386-a0cb-4313-86fc-8a82c931598a-kube-api-access-z8rvn" pod "34fa7386-a0cb-4313-86fc-8a82c931598a" (UID: "34fa7386-a0cb-4313-86fc-8a82c931598a"). InnerVolumeSpecName "kube-api-access-z8rvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:38.345559 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:38.345524 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8rvn\" (UniqueName: \"kubernetes.io/projected/34fa7386-a0cb-4313-86fc-8a82c931598a-kube-api-access-z8rvn\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:38.447363 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:38.447253 4503 generic.go:356] "Generic (PLEG): container finished" podID="34fa7386-a0cb-4313-86fc-8a82c931598a" containerID="b768bdbc0e3bf9b47cbfbbc68858f4dc7228102f4db076ef8ce6bdd1b925252d" exitCode=0 Oct 30 13:11:38.447363 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:38.447322 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" Oct 30 13:11:38.447363 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:38.447328 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" event={"ID":"34fa7386-a0cb-4313-86fc-8a82c931598a","Type":"ContainerDied","Data":"b768bdbc0e3bf9b47cbfbbc68858f4dc7228102f4db076ef8ce6bdd1b925252d"} Oct 30 13:11:38.447648 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:38.447406 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k" event={"ID":"34fa7386-a0cb-4313-86fc-8a82c931598a","Type":"ContainerDied","Data":"45fdb0782c5a04cead7773e9b668126ebe2779a689a94992266e218804433081"} Oct 30 13:11:38.447648 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:38.447430 4503 scope.go:122] "RemoveContainer" containerID="b768bdbc0e3bf9b47cbfbbc68858f4dc7228102f4db076ef8ce6bdd1b925252d" Oct 30 13:11:38.460115 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:38.460081 4503 scope.go:122] "RemoveContainer" containerID="b768bdbc0e3bf9b47cbfbbc68858f4dc7228102f4db076ef8ce6bdd1b925252d" Oct 30 13:11:38.460724 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:11:38.460683 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b768bdbc0e3bf9b47cbfbbc68858f4dc7228102f4db076ef8ce6bdd1b925252d\": container with ID starting with b768bdbc0e3bf9b47cbfbbc68858f4dc7228102f4db076ef8ce6bdd1b925252d not found: ID does not exist" containerID="b768bdbc0e3bf9b47cbfbbc68858f4dc7228102f4db076ef8ce6bdd1b925252d" Oct 30 13:11:38.460862 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:38.460731 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b768bdbc0e3bf9b47cbfbbc68858f4dc7228102f4db076ef8ce6bdd1b925252d"} err="failed to get container status \"b768bdbc0e3bf9b47cbfbbc68858f4dc7228102f4db076ef8ce6bdd1b925252d\": rpc error: code = NotFound desc = could not find container \"b768bdbc0e3bf9b47cbfbbc68858f4dc7228102f4db076ef8ce6bdd1b925252d\": container with ID starting with b768bdbc0e3bf9b47cbfbbc68858f4dc7228102f4db076ef8ce6bdd1b925252d not found: ID does not exist" Oct 30 13:11:38.503048 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:38.503001 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k"] Oct 30 13:11:38.508275 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:38.508251 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3048f9-78dbf57d76-wgf4k"] Oct 30 13:11:38.733503 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:38.733419 4503 patch_prober.go:29] interesting pod/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:11:38.733503 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:38.733478 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" podUID="4151e97e-c25a-49af-8268-baa1c493ee4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:11:40.392792 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:40.392743 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34fa7386-a0cb-4313-86fc-8a82c931598a" path="/var/lib/kubelet/pods/34fa7386-a0cb-4313-86fc-8a82c931598a/volumes" Oct 30 13:11:42.519319 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.519276 4503 generic.go:356] "Generic (PLEG): container finished" podID="e0332beb-0b14-493b-9a37-4de43236c083" containerID="b71c4743a919c38ffd47173db043f957593e20052b590a28054cf60977388916" exitCode=0 Oct 30 13:11:42.519919 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.519432 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" event={"ID":"e0332beb-0b14-493b-9a37-4de43236c083","Type":"ContainerDied","Data":"b71c4743a919c38ffd47173db043f957593e20052b590a28054cf60977388916"} Oct 30 13:11:42.519919 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.519487 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" event={"ID":"e0332beb-0b14-493b-9a37-4de43236c083","Type":"ContainerDied","Data":"b658df5da5752e614cf08cb6a69b9e8075096b9c859c62e84b2495ef901eb9e0"} Oct 30 13:11:42.519919 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.519512 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b658df5da5752e614cf08cb6a69b9e8075096b9c859c62e84b2495ef901eb9e0" Oct 30 13:11:42.519919 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.519841 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:11:42.532078 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.532020 4503 generic.go:356] "Generic (PLEG): container finished" podID="70f0f890-bb6a-4627-9487-90741d34337c" containerID="d50382081c56a2f580490f8b8cbd3a4025d957d1c04889923a3389a2cc6b4573" exitCode=0 Oct 30 13:11:42.532252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.532132 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" event={"ID":"70f0f890-bb6a-4627-9487-90741d34337c","Type":"ContainerDied","Data":"d50382081c56a2f580490f8b8cbd3a4025d957d1c04889923a3389a2cc6b4573"} Oct 30 13:11:42.532252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.532193 4503 scope.go:122] "RemoveContainer" containerID="509359a3e740377c45d71cd18c332ef3fc4ee611c47e789912fb7006ab378970" Oct 30 13:11:42.535240 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.535216 4503 generic.go:356] "Generic (PLEG): container finished" podID="4151e97e-c25a-49af-8268-baa1c493ee4a" containerID="519b9a429ca3e6daf718e2c698e775b7b40f6e08bbcb820cd1ddbb85934e41ea" exitCode=0 Oct 30 13:11:42.535326 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.535314 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" event={"ID":"4151e97e-c25a-49af-8268-baa1c493ee4a","Type":"ContainerDied","Data":"519b9a429ca3e6daf718e2c698e775b7b40f6e08bbcb820cd1ddbb85934e41ea"} Oct 30 13:11:42.538111 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.537896 4503 generic.go:356] "Generic (PLEG): container finished" podID="f6a2a7df-fe4f-4055-9de6-b1acaae0db54" containerID="593a2b8b4297111030ace0025a29f43d9f5ccb30aa1e5ee508db2dd9fcc24cd4" exitCode=0 Oct 30 13:11:42.538111 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.537984 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" event={"ID":"f6a2a7df-fe4f-4055-9de6-b1acaae0db54","Type":"ContainerDied","Data":"593a2b8b4297111030ace0025a29f43d9f5ccb30aa1e5ee508db2dd9fcc24cd4"} Oct 30 13:11:42.549544 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.549462 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:11:42.551259 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.551031 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-workload-certs\") pod \"e0332beb-0b14-493b-9a37-4de43236c083\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " Oct 30 13:11:42.551259 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.551099 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-istio-envoy\") pod \"e0332beb-0b14-493b-9a37-4de43236c083\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " Oct 30 13:11:42.551259 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.551154 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-istio-data\") pod \"e0332beb-0b14-493b-9a37-4de43236c083\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " Oct 30 13:11:42.551995 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.551610 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-workload-socket\") pod \"e0332beb-0b14-493b-9a37-4de43236c083\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " Oct 30 13:11:42.551995 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.551656 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0332beb-0b14-493b-9a37-4de43236c083-kube-api-access-rm6qd\" (UniqueName: \"kubernetes.io/projected/e0332beb-0b14-493b-9a37-4de43236c083-kube-api-access-rm6qd\") pod \"e0332beb-0b14-493b-9a37-4de43236c083\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " Oct 30 13:11:42.551995 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.551743 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0332beb-0b14-493b-9a37-4de43236c083-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0332beb-0b14-493b-9a37-4de43236c083-istiod-ca-cert\") pod \"e0332beb-0b14-493b-9a37-4de43236c083\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " Oct 30 13:11:42.551995 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.551802 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-credential-socket\") pod \"e0332beb-0b14-493b-9a37-4de43236c083\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " Oct 30 13:11:42.551995 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.551855 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0332beb-0b14-493b-9a37-4de43236c083-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0332beb-0b14-493b-9a37-4de43236c083-istio-podinfo\") pod \"e0332beb-0b14-493b-9a37-4de43236c083\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " Oct 30 13:11:42.551995 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.551900 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0332beb-0b14-493b-9a37-4de43236c083-istio-token\" (UniqueName: \"kubernetes.io/projected/e0332beb-0b14-493b-9a37-4de43236c083-istio-token\") pod \"e0332beb-0b14-493b-9a37-4de43236c083\" (UID: \"e0332beb-0b14-493b-9a37-4de43236c083\") " Oct 30 13:11:42.552328 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.552015 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-workload-certs" pod "e0332beb-0b14-493b-9a37-4de43236c083" (UID: "e0332beb-0b14-493b-9a37-4de43236c083"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.552642 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.552620 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0332beb-0b14-493b-9a37-4de43236c083-istiod-ca-cert" pod "e0332beb-0b14-493b-9a37-4de43236c083" (UID: "e0332beb-0b14-493b-9a37-4de43236c083"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:42.553058 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.552988 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-credential-socket" pod "e0332beb-0b14-493b-9a37-4de43236c083" (UID: "e0332beb-0b14-493b-9a37-4de43236c083"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.553406 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.553384 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.553505 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.553494 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0332beb-0b14-493b-9a37-4de43236c083-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.553592 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.553581 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.553965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.553929 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-workload-socket" pod "e0332beb-0b14-493b-9a37-4de43236c083" (UID: "e0332beb-0b14-493b-9a37-4de43236c083"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.556059 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.556038 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-istio-data" pod "e0332beb-0b14-493b-9a37-4de43236c083" (UID: "e0332beb-0b14-493b-9a37-4de43236c083"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.557991 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.557486 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0332beb-0b14-493b-9a37-4de43236c083-istio-podinfo" pod "e0332beb-0b14-493b-9a37-4de43236c083" (UID: "e0332beb-0b14-493b-9a37-4de43236c083"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:42.567355 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.567309 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0332beb-0b14-493b-9a37-4de43236c083-kube-api-access-rm6qd" pod "e0332beb-0b14-493b-9a37-4de43236c083" (UID: "e0332beb-0b14-493b-9a37-4de43236c083"). InnerVolumeSpecName "kube-api-access-rm6qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:42.569620 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.569577 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0332beb-0b14-493b-9a37-4de43236c083-istio-token" pod "e0332beb-0b14-493b-9a37-4de43236c083" (UID: "e0332beb-0b14-493b-9a37-4de43236c083"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:42.570416 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.570369 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-istio-envoy" pod "e0332beb-0b14-493b-9a37-4de43236c083" (UID: "e0332beb-0b14-493b-9a37-4de43236c083"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.613131 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.613109 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:11:42.620836 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.620809 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:11:42.654987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.654325 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-workload-certs\") pod \"70f0f890-bb6a-4627-9487-90741d34337c\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " Oct 30 13:11:42.654987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.654378 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70f0f890-bb6a-4627-9487-90741d34337c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f0f890-bb6a-4627-9487-90741d34337c-istio-podinfo\") pod \"70f0f890-bb6a-4627-9487-90741d34337c\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " Oct 30 13:11:42.654987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.654416 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-envoy\") pod \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " Oct 30 13:11:42.654987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.654449 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f0f890-bb6a-4627-9487-90741d34337c-kube-api-access-x6tpm\" (UniqueName: \"kubernetes.io/projected/70f0f890-bb6a-4627-9487-90741d34337c-kube-api-access-x6tpm\") pod \"70f0f890-bb6a-4627-9487-90741d34337c\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " Oct 30 13:11:42.654987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.654505 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-workload-socket\") pod \"4151e97e-c25a-49af-8268-baa1c493ee4a\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " Oct 30 13:11:42.654987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.654535 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-workload-socket\") pod \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " Oct 30 13:11:42.654987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.654595 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-istio-data\") pod \"70f0f890-bb6a-4627-9487-90741d34337c\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " Oct 30 13:11:42.654987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.654650 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4151e97e-c25a-49af-8268-baa1c493ee4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4151e97e-c25a-49af-8268-baa1c493ee4a-istiod-ca-cert\") pod \"4151e97e-c25a-49af-8268-baa1c493ee4a\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " Oct 30 13:11:42.654987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.654744 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-workload-socket\") pod \"70f0f890-bb6a-4627-9487-90741d34337c\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " Oct 30 13:11:42.654987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.654786 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-istio-envoy\") pod \"70f0f890-bb6a-4627-9487-90741d34337c\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " Oct 30 13:11:42.654987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.654816 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-workload-certs\") pod \"4151e97e-c25a-49af-8268-baa1c493ee4a\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " Oct 30 13:11:42.654987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.654852 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-podinfo\") pod \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " Oct 30 13:11:42.654987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.654884 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-data\") pod \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " Oct 30 13:11:42.654987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.654911 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-kube-api-access-hzkxn\" (UniqueName: \"kubernetes.io/projected/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-kube-api-access-hzkxn\") pod \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " Oct 30 13:11:42.654987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.654963 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-workload-certs\") pod \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " Oct 30 13:11:42.654987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.654992 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-credential-socket\") pod \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " Oct 30 13:11:42.655952 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.655018 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f0f890-bb6a-4627-9487-90741d34337c-istio-token\" (UniqueName: \"kubernetes.io/projected/70f0f890-bb6a-4627-9487-90741d34337c-istio-token\") pod \"70f0f890-bb6a-4627-9487-90741d34337c\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " Oct 30 13:11:42.655952 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.655079 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-credential-socket\") pod \"70f0f890-bb6a-4627-9487-90741d34337c\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " Oct 30 13:11:42.655952 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.655117 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f0f890-bb6a-4627-9487-90741d34337c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f0f890-bb6a-4627-9487-90741d34337c-istiod-ca-cert\") pod \"70f0f890-bb6a-4627-9487-90741d34337c\" (UID: \"70f0f890-bb6a-4627-9487-90741d34337c\") " Oct 30 13:11:42.655952 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.655158 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-data\") pod \"4151e97e-c25a-49af-8268-baa1c493ee4a\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " Oct 30 13:11:42.655952 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.655202 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-token\" (UniqueName: \"kubernetes.io/projected/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-token\") pod \"4151e97e-c25a-49af-8268-baa1c493ee4a\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " Oct 30 13:11:42.655952 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.655249 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-podinfo\") pod \"4151e97e-c25a-49af-8268-baa1c493ee4a\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " Oct 30 13:11:42.655952 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.655304 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istiod-ca-cert\") pod \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " Oct 30 13:11:42.655952 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.655351 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-token\" (UniqueName: \"kubernetes.io/projected/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-token\") pod \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\" (UID: \"f6a2a7df-fe4f-4055-9de6-b1acaae0db54\") " Oct 30 13:11:42.655952 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.655405 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-credential-socket\") pod \"4151e97e-c25a-49af-8268-baa1c493ee4a\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " Oct 30 13:11:42.655952 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.655439 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4151e97e-c25a-49af-8268-baa1c493ee4a-kube-api-access-d4qnt\" (UniqueName: \"kubernetes.io/projected/4151e97e-c25a-49af-8268-baa1c493ee4a-kube-api-access-d4qnt\") pod \"4151e97e-c25a-49af-8268-baa1c493ee4a\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " Oct 30 13:11:42.655952 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.655518 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-envoy\") pod \"4151e97e-c25a-49af-8268-baa1c493ee4a\" (UID: \"4151e97e-c25a-49af-8268-baa1c493ee4a\") " Oct 30 13:11:42.674143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.656087 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-data" pod "f6a2a7df-fe4f-4055-9de6-b1acaae0db54" (UID: "f6a2a7df-fe4f-4055-9de6-b1acaae0db54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.674143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.656323 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-workload-certs" pod "70f0f890-bb6a-4627-9487-90741d34337c" (UID: "70f0f890-bb6a-4627-9487-90741d34337c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.674143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.656821 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-data" pod "4151e97e-c25a-49af-8268-baa1c493ee4a" (UID: "4151e97e-c25a-49af-8268-baa1c493ee4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.674143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.657320 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-istio-data" pod "70f0f890-bb6a-4627-9487-90741d34337c" (UID: "70f0f890-bb6a-4627-9487-90741d34337c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.674143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.657733 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-credential-socket" pod "4151e97e-c25a-49af-8268-baa1c493ee4a" (UID: "4151e97e-c25a-49af-8268-baa1c493ee4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.674143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.657764 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istiod-ca-cert" pod "f6a2a7df-fe4f-4055-9de6-b1acaae0db54" (UID: "f6a2a7df-fe4f-4055-9de6-b1acaae0db54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:42.674143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.658254 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-workload-socket" pod "4151e97e-c25a-49af-8268-baa1c493ee4a" (UID: "4151e97e-c25a-49af-8268-baa1c493ee4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.674143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.661170 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4151e97e-c25a-49af-8268-baa1c493ee4a-istiod-ca-cert" pod "4151e97e-c25a-49af-8268-baa1c493ee4a" (UID: "4151e97e-c25a-49af-8268-baa1c493ee4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:42.674143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.661386 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-workload-socket" pod "70f0f890-bb6a-4627-9487-90741d34337c" (UID: "70f0f890-bb6a-4627-9487-90741d34337c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.674143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.664002 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-workload-certs" pod "4151e97e-c25a-49af-8268-baa1c493ee4a" (UID: "4151e97e-c25a-49af-8268-baa1c493ee4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.674143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.673559 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-workload-certs" pod "f6a2a7df-fe4f-4055-9de6-b1acaae0db54" (UID: "f6a2a7df-fe4f-4055-9de6-b1acaae0db54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.674143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.673719 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-credential-socket" pod "f6a2a7df-fe4f-4055-9de6-b1acaae0db54" (UID: "f6a2a7df-fe4f-4055-9de6-b1acaae0db54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.674143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.673804 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f0f890-bb6a-4627-9487-90741d34337c-istio-token" pod "70f0f890-bb6a-4627-9487-90741d34337c" (UID: "70f0f890-bb6a-4627-9487-90741d34337c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:42.674911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.674595 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.674911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.674624 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm6qd\" (UniqueName: \"kubernetes.io/projected/e0332beb-0b14-493b-9a37-4de43236c083-kube-api-access-rm6qd\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.674911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.674639 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.674911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.674653 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.674911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.674666 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.674911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.674678 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0332beb-0b14-493b-9a37-4de43236c083-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.674911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.674692 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0332beb-0b14-493b-9a37-4de43236c083-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.674911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.674705 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.674911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.674719 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.674911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.674731 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.674911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.674744 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0332beb-0b14-493b-9a37-4de43236c083-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.674911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.674756 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.674911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.674769 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4151e97e-c25a-49af-8268-baa1c493ee4a-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.674911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.674782 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.674911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.674795 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.674911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.674808 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.674911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.674821 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.674911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.674834 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.674911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.674846 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f0f890-bb6a-4627-9487-90741d34337c-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.675930 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.675082 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-credential-socket" pod "70f0f890-bb6a-4627-9487-90741d34337c" (UID: "70f0f890-bb6a-4627-9487-90741d34337c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.675930 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.675345 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f0f890-bb6a-4627-9487-90741d34337c-istiod-ca-cert" pod "70f0f890-bb6a-4627-9487-90741d34337c" (UID: "70f0f890-bb6a-4627-9487-90741d34337c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:42.675930 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.675575 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-workload-socket" pod "f6a2a7df-fe4f-4055-9de6-b1acaae0db54" (UID: "f6a2a7df-fe4f-4055-9de6-b1acaae0db54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.679241 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.679199 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-podinfo" pod "f6a2a7df-fe4f-4055-9de6-b1acaae0db54" (UID: "f6a2a7df-fe4f-4055-9de6-b1acaae0db54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:42.679347 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.679239 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f0f890-bb6a-4627-9487-90741d34337c-kube-api-access-x6tpm" pod "70f0f890-bb6a-4627-9487-90741d34337c" (UID: "70f0f890-bb6a-4627-9487-90741d34337c"). InnerVolumeSpecName "kube-api-access-x6tpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:42.680123 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.680097 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-podinfo" pod "4151e97e-c25a-49af-8268-baa1c493ee4a" (UID: "4151e97e-c25a-49af-8268-baa1c493ee4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:42.680234 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.680217 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-token" pod "f6a2a7df-fe4f-4055-9de6-b1acaae0db54" (UID: "f6a2a7df-fe4f-4055-9de6-b1acaae0db54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:42.680359 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.680331 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-istio-envoy" pod "70f0f890-bb6a-4627-9487-90741d34337c" (UID: "70f0f890-bb6a-4627-9487-90741d34337c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.680443 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.680426 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-token" pod "4151e97e-c25a-49af-8268-baa1c493ee4a" (UID: "4151e97e-c25a-49af-8268-baa1c493ee4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:42.680552 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.680536 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70f0f890-bb6a-4627-9487-90741d34337c-istio-podinfo" pod "70f0f890-bb6a-4627-9487-90741d34337c" (UID: "70f0f890-bb6a-4627-9487-90741d34337c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:42.682172 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.682144 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-envoy" pod "4151e97e-c25a-49af-8268-baa1c493ee4a" (UID: "4151e97e-c25a-49af-8268-baa1c493ee4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.684106 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.684078 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-envoy" pod "f6a2a7df-fe4f-4055-9de6-b1acaae0db54" (UID: "f6a2a7df-fe4f-4055-9de6-b1acaae0db54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:42.684205 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.684124 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4151e97e-c25a-49af-8268-baa1c493ee4a-kube-api-access-d4qnt" pod "4151e97e-c25a-49af-8268-baa1c493ee4a" (UID: "4151e97e-c25a-49af-8268-baa1c493ee4a"). InnerVolumeSpecName "kube-api-access-d4qnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:42.684265 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.684202 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-kube-api-access-hzkxn" pod "f6a2a7df-fe4f-4055-9de6-b1acaae0db54" (UID: "f6a2a7df-fe4f-4055-9de6-b1acaae0db54"). InnerVolumeSpecName "kube-api-access-hzkxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:42.776060 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.775846 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.776060 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.775872 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.776060 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.775881 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzkxn\" (UniqueName: \"kubernetes.io/projected/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-kube-api-access-hzkxn\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.776060 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.775890 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f0f890-bb6a-4627-9487-90741d34337c-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.776060 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.775899 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f0f890-bb6a-4627-9487-90741d34337c-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.776060 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.775907 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.776060 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.775914 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.776060 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.775923 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.776060 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.775932 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4qnt\" (UniqueName: \"kubernetes.io/projected/4151e97e-c25a-49af-8268-baa1c493ee4a-kube-api-access-d4qnt\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.776060 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.775963 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4151e97e-c25a-49af-8268-baa1c493ee4a-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.776060 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.775972 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f0f890-bb6a-4627-9487-90741d34337c-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.776060 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.775980 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.776060 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.775988 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6tpm\" (UniqueName: \"kubernetes.io/projected/70f0f890-bb6a-4627-9487-90741d34337c-kube-api-access-x6tpm\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:42.776060 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:42.775996 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a7df-fe4f-4055-9de6-b1acaae0db54-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:43.549307 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:43.549207 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" Oct 30 13:11:43.568631 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:43.568586 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6" event={"ID":"70f0f890-bb6a-4627-9487-90741d34337c","Type":"ContainerDied","Data":"b765045eed14df8e8f5fc118b0e7e1a08a18eec691ca6bb6956a86681e6baa4b"} Oct 30 13:11:43.568968 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:43.568924 4503 scope.go:122] "RemoveContainer" containerID="d50382081c56a2f580490f8b8cbd3a4025d957d1c04889923a3389a2cc6b4573" Oct 30 13:11:43.574583 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:43.574528 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" event={"ID":"4151e97e-c25a-49af-8268-baa1c493ee4a","Type":"ContainerDied","Data":"87aaee5368362c1985d0fcbd1de3734edbbea6580a117eee5df78ff621ed5874"} Oct 30 13:11:43.574867 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:43.574844 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj" Oct 30 13:11:43.606221 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:43.606191 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv" Oct 30 13:11:43.606828 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:43.606418 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" Oct 30 13:11:43.607783 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:43.606444 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj" event={"ID":"f6a2a7df-fe4f-4055-9de6-b1acaae0db54","Type":"ContainerDied","Data":"798d75960333dd75aecf60e2924b2c32c823772a8c31a57399f23725cd9c1b86"} Oct 30 13:11:43.631827 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:43.631803 4503 scope.go:122] "RemoveContainer" containerID="519b9a429ca3e6daf718e2c698e775b7b40f6e08bbcb820cd1ddbb85934e41ea" Oct 30 13:11:43.643593 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:43.643566 4503 scope.go:122] "RemoveContainer" containerID="593a2b8b4297111030ace0025a29f43d9f5ccb30aa1e5ee508db2dd9fcc24cd4" Oct 30 13:11:43.684001 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:43.683966 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj"] Oct 30 13:11:43.700283 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:43.700254 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-h7xvj"] Oct 30 13:11:43.736777 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:43.736747 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv"] Oct 30 13:11:43.744801 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:43.744771 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-vgrzv"] Oct 30 13:11:43.782142 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:43.782109 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj"] Oct 30 13:11:43.791292 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:43.791254 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-47xxj"] Oct 30 13:11:43.825109 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:43.825024 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6"] Oct 30 13:11:43.832280 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:43.832250 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42ee9d-openshift-default-74dd7567f9-lpkb6"] Oct 30 13:11:44.378316 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:44.377641 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4151e97e-c25a-49af-8268-baa1c493ee4a" path="/var/lib/kubelet/pods/4151e97e-c25a-49af-8268-baa1c493ee4a/volumes" Oct 30 13:11:44.378580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:44.378540 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70f0f890-bb6a-4627-9487-90741d34337c" path="/var/lib/kubelet/pods/70f0f890-bb6a-4627-9487-90741d34337c/volumes" Oct 30 13:11:44.381291 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:44.379436 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0332beb-0b14-493b-9a37-4de43236c083" path="/var/lib/kubelet/pods/e0332beb-0b14-493b-9a37-4de43236c083/volumes" Oct 30 13:11:44.381291 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:44.380680 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a2a7df-fe4f-4055-9de6-b1acaae0db54" path="/var/lib/kubelet/pods/f6a2a7df-fe4f-4055-9de6-b1acaae0db54/volumes" Oct 30 13:11:44.793188 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:44.793147 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x"] Oct 30 13:11:44.793664 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:44.793402 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" podUID="391723a7-7e4d-4727-af52-abe87e1ab8ff" containerName="maas-api" containerID="cri-o://69c5b8eb8be087810af9de827569493ae7d012a2d4b2ecf0cf425de63ee9beb7" gracePeriod=30 Oct 30 13:11:45.242889 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.242866 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" Oct 30 13:11:45.341700 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.341662 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391723a7-7e4d-4727-af52-abe87e1ab8ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/391723a7-7e4d-4727-af52-abe87e1ab8ff-metrics-certs\") pod \"391723a7-7e4d-4727-af52-abe87e1ab8ff\" (UID: \"391723a7-7e4d-4727-af52-abe87e1ab8ff\") " Oct 30 13:11:45.341868 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.341725 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391723a7-7e4d-4727-af52-abe87e1ab8ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/391723a7-7e4d-4727-af52-abe87e1ab8ff-openshift-service-ca\") pod \"391723a7-7e4d-4727-af52-abe87e1ab8ff\" (UID: \"391723a7-7e4d-4727-af52-abe87e1ab8ff\") " Oct 30 13:11:45.341868 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.341850 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391723a7-7e4d-4727-af52-abe87e1ab8ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/391723a7-7e4d-4727-af52-abe87e1ab8ff-maas-api-tls\") pod \"391723a7-7e4d-4727-af52-abe87e1ab8ff\" (UID: \"391723a7-7e4d-4727-af52-abe87e1ab8ff\") " Oct 30 13:11:45.342080 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.342052 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/391723a7-7e4d-4727-af52-abe87e1ab8ff-kube-api-access-qk6m7\" (UniqueName: \"kubernetes.io/projected/391723a7-7e4d-4727-af52-abe87e1ab8ff-kube-api-access-qk6m7\") pod \"391723a7-7e4d-4727-af52-abe87e1ab8ff\" (UID: \"391723a7-7e4d-4727-af52-abe87e1ab8ff\") " Oct 30 13:11:45.342213 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.342137 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391723a7-7e4d-4727-af52-abe87e1ab8ff-openshift-service-ca" pod "391723a7-7e4d-4727-af52-abe87e1ab8ff" (UID: "391723a7-7e4d-4727-af52-abe87e1ab8ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:45.342541 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.342524 4503 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/391723a7-7e4d-4727-af52-abe87e1ab8ff-openshift-service-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:45.343915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.343893 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391723a7-7e4d-4727-af52-abe87e1ab8ff-metrics-certs" pod "391723a7-7e4d-4727-af52-abe87e1ab8ff" (UID: "391723a7-7e4d-4727-af52-abe87e1ab8ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:11:45.344139 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.344118 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391723a7-7e4d-4727-af52-abe87e1ab8ff-maas-api-tls" pod "391723a7-7e4d-4727-af52-abe87e1ab8ff" (UID: "391723a7-7e4d-4727-af52-abe87e1ab8ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:11:45.344257 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.344235 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391723a7-7e4d-4727-af52-abe87e1ab8ff-kube-api-access-qk6m7" pod "391723a7-7e4d-4727-af52-abe87e1ab8ff" (UID: "391723a7-7e4d-4727-af52-abe87e1ab8ff"). InnerVolumeSpecName "kube-api-access-qk6m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:45.444017 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.443906 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/391723a7-7e4d-4727-af52-abe87e1ab8ff-maas-api-tls\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:45.444017 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.443978 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk6m7\" (UniqueName: \"kubernetes.io/projected/391723a7-7e4d-4727-af52-abe87e1ab8ff-kube-api-access-qk6m7\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:45.444017 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.443996 4503 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/391723a7-7e4d-4727-af52-abe87e1ab8ff-metrics-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:45.648050 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.644100 4503 generic.go:356] "Generic (PLEG): container finished" podID="391723a7-7e4d-4727-af52-abe87e1ab8ff" containerID="69c5b8eb8be087810af9de827569493ae7d012a2d4b2ecf0cf425de63ee9beb7" exitCode=0 Oct 30 13:11:45.648050 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.644329 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" event={"ID":"391723a7-7e4d-4727-af52-abe87e1ab8ff","Type":"ContainerDied","Data":"69c5b8eb8be087810af9de827569493ae7d012a2d4b2ecf0cf425de63ee9beb7"} Oct 30 13:11:45.648050 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.644379 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" event={"ID":"391723a7-7e4d-4727-af52-abe87e1ab8ff","Type":"ContainerDied","Data":"0e103c1588543c6742dff3314a9d711552366e7545c9b6da182d5d2e92a9cd96"} Oct 30 13:11:45.648050 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.644412 4503 scope.go:122] "RemoveContainer" containerID="69c5b8eb8be087810af9de827569493ae7d012a2d4b2ecf0cf425de63ee9beb7" Oct 30 13:11:45.648050 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.644520 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x" Oct 30 13:11:45.695965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.695565 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm"] Oct 30 13:11:45.695965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.695812 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" podUID="04fb46b3-a555-41ab-a7be-f57ed8c9deaf" containerName="istio-proxy" containerID="cri-o://132f90033fd2e255d137cf3db1a4d4dcdc2f86c05f2e82afa310d0ecbb0379f2" gracePeriod=30 Oct 30 13:11:45.704973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.704126 4503 scope.go:122] "RemoveContainer" containerID="69c5b8eb8be087810af9de827569493ae7d012a2d4b2ecf0cf425de63ee9beb7" Oct 30 13:11:45.704973 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:11:45.704522 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c5b8eb8be087810af9de827569493ae7d012a2d4b2ecf0cf425de63ee9beb7\": container with ID starting with 69c5b8eb8be087810af9de827569493ae7d012a2d4b2ecf0cf425de63ee9beb7 not found: ID does not exist" containerID="69c5b8eb8be087810af9de827569493ae7d012a2d4b2ecf0cf425de63ee9beb7" Oct 30 13:11:45.704973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.704559 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c5b8eb8be087810af9de827569493ae7d012a2d4b2ecf0cf425de63ee9beb7"} err="failed to get container status \"69c5b8eb8be087810af9de827569493ae7d012a2d4b2ecf0cf425de63ee9beb7\": rpc error: code = NotFound desc = could not find container \"69c5b8eb8be087810af9de827569493ae7d012a2d4b2ecf0cf425de63ee9beb7\": container with ID starting with 69c5b8eb8be087810af9de827569493ae7d012a2d4b2ecf0cf425de63ee9beb7 not found: ID does not exist" Oct 30 13:11:45.732092 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.732058 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x"] Oct 30 13:11:45.739180 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:45.739155 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0fd5fcba-649d78b578-l8c4x"] Oct 30 13:11:46.331338 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:46.331300 4503 patch_prober.go:29] interesting pod/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:11:46.334095 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:46.334060 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" podUID="04fb46b3-a555-41ab-a7be-f57ed8c9deaf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:11:46.384009 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:46.383970 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="391723a7-7e4d-4727-af52-abe87e1ab8ff" path="/var/lib/kubelet/pods/391723a7-7e4d-4727-af52-abe87e1ab8ff/volumes" Oct 30 13:11:47.432388 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:47.432351 4503 scope.go:122] "RemoveContainer" containerID="a627a0be781883afd17a75bd0f9add6d3f6207262e6f300161418c626d7afc23" Oct 30 13:11:50.587256 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.587211 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9"] Oct 30 13:11:50.588191 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588169 4503 cpu_manager.go:414] "containerMap: removing container" podUID="70f0f890-bb6a-4627-9487-90741d34337c" containerName="istio-proxy" Oct 30 13:11:50.588257 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588197 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f0f890-bb6a-4627-9487-90741d34337c" containerName="istio-proxy" Oct 30 13:11:50.588257 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588228 4503 cpu_manager.go:414] "containerMap: removing container" podUID="e0332beb-0b14-493b-9a37-4de43236c083" containerName="istio-proxy" Oct 30 13:11:50.588257 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588236 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0332beb-0b14-493b-9a37-4de43236c083" containerName="istio-proxy" Oct 30 13:11:50.588257 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588246 4503 cpu_manager.go:414] "containerMap: removing container" podUID="391723a7-7e4d-4727-af52-abe87e1ab8ff" containerName="maas-api" Oct 30 13:11:50.588257 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588254 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="391723a7-7e4d-4727-af52-abe87e1ab8ff" containerName="maas-api" Oct 30 13:11:50.588412 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588273 4503 cpu_manager.go:414] "containerMap: removing container" podUID="f6a2a7df-fe4f-4055-9de6-b1acaae0db54" containerName="istio-proxy" Oct 30 13:11:50.588412 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588283 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a2a7df-fe4f-4055-9de6-b1acaae0db54" containerName="istio-proxy" Oct 30 13:11:50.588412 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588318 4503 cpu_manager.go:414] "containerMap: removing container" podUID="34fa7386-a0cb-4313-86fc-8a82c931598a" containerName="payload-processing" Oct 30 13:11:50.588412 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588325 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34fa7386-a0cb-4313-86fc-8a82c931598a" containerName="payload-processing" Oct 30 13:11:50.588412 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588353 4503 cpu_manager.go:414] "containerMap: removing container" podUID="0549ac29-5939-4f6c-aced-091b158f4afd" containerName="istio-proxy" Oct 30 13:11:50.588412 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588360 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0549ac29-5939-4f6c-aced-091b158f4afd" containerName="istio-proxy" Oct 30 13:11:50.588412 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588374 4503 cpu_manager.go:414] "containerMap: removing container" podUID="803ac7a6-439b-4634-a116-5e16c2595f42" containerName="payload-pre-processing" Oct 30 13:11:50.588412 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588381 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="803ac7a6-439b-4634-a116-5e16c2595f42" containerName="payload-pre-processing" Oct 30 13:11:50.588412 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588400 4503 cpu_manager.go:414] "containerMap: removing container" podUID="4151e97e-c25a-49af-8268-baa1c493ee4a" containerName="istio-proxy" Oct 30 13:11:50.588412 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588408 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4151e97e-c25a-49af-8268-baa1c493ee4a" containerName="istio-proxy" Oct 30 13:11:50.588683 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588559 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f0f890-bb6a-4627-9487-90741d34337c" containerName="istio-proxy" Oct 30 13:11:50.588683 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588575 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="34fa7386-a0cb-4313-86fc-8a82c931598a" containerName="payload-processing" Oct 30 13:11:50.588683 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588592 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="0549ac29-5939-4f6c-aced-091b158f4afd" containerName="istio-proxy" Oct 30 13:11:50.588683 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588601 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="803ac7a6-439b-4634-a116-5e16c2595f42" containerName="payload-pre-processing" Oct 30 13:11:50.588683 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588614 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="4151e97e-c25a-49af-8268-baa1c493ee4a" containerName="istio-proxy" Oct 30 13:11:50.588683 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588624 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f0f890-bb6a-4627-9487-90741d34337c" containerName="istio-proxy" Oct 30 13:11:50.588683 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588635 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="391723a7-7e4d-4727-af52-abe87e1ab8ff" containerName="maas-api" Oct 30 13:11:50.588683 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588645 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0332beb-0b14-493b-9a37-4de43236c083" containerName="istio-proxy" Oct 30 13:11:50.588683 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.588654 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a2a7df-fe4f-4055-9de6-b1acaae0db54" containerName="istio-proxy" Oct 30 13:11:50.595164 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.595120 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.633999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.630638 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-ac0c58-openshift-default-dockercfg-5mbd5\"" Oct 30 13:11:50.645389 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.644604 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9"] Oct 30 13:11:50.713773 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.693801 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-workload-certs\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.713773 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.693847 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-podinfo\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.713773 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.710651 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-data\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.713773 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.710728 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-token\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.713773 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.710807 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-workload-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.713773 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.710887 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istiod-ca-cert\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.713773 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.710925 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdwg8\" (UniqueName: \"kubernetes.io/projected/692ff35a-d45a-4a6e-80f6-92511e76e7fd-kube-api-access-bdwg8\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.713773 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.711024 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-credential-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.713773 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.711121 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-envoy\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.781064 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.781017 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp"] Oct 30 13:11:50.814999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.796254 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" podUID="81a74b8e-b1f2-467b-bb2c-6b0029e464f4" containerName="istio-proxy" containerID="cri-o://4af7d550e92bda682187fd81e5821e501d48dec4dd0f744701ee746d1e403f05" gracePeriod=30 Oct 30 13:11:50.814999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.812684 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-data\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.814999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.812762 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-token\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.814999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.812830 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-workload-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.814999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.812896 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istiod-ca-cert\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.814999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.812932 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdwg8\" (UniqueName: \"kubernetes.io/projected/692ff35a-d45a-4a6e-80f6-92511e76e7fd-kube-api-access-bdwg8\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.814999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.813018 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-credential-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.814999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.813101 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-envoy\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.814999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.813145 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-workload-certs\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.814999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.813173 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-podinfo\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.814999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.814562 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-credential-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.816491 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.816454 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-data\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.816756 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.816731 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-workload-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.819413 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.819382 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-workload-certs\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.828961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.820734 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istiod-ca-cert\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.828961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.824838 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6"] Oct 30 13:11:50.828961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.825098 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" podUID="25ebe375-c4db-41eb-bc87-3cb9d8e0853b" containerName="istio-proxy" containerID="cri-o://3db6d80a5e252a56b71a9a0666c47917154ee121516f7df5b9e48d6449aa798a" gracePeriod=30 Oct 30 13:11:50.830413 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.830381 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m"] Oct 30 13:11:50.830694 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.830647 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" podUID="788f38f0-c8b1-461b-969e-8d171ad33149" containerName="istio-proxy" containerID="cri-o://b4663661b422579d5b7c09addfc9808b6aad52711328bc82dc422abd2b326947" gracePeriod=30 Oct 30 13:11:50.852593 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.842993 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdwg8\" (UniqueName: \"kubernetes.io/projected/692ff35a-d45a-4a6e-80f6-92511e76e7fd-kube-api-access-bdwg8\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.858980 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.855858 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-token\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.875575 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.866446 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-podinfo\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:50.967962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:50.964351 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-envoy\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:51.216957 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.216203 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:51.788999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.765409 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:11:51.788999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.769528 4503 generic.go:356] "Generic (PLEG): container finished" podID="04fb46b3-a555-41ab-a7be-f57ed8c9deaf" containerID="132f90033fd2e255d137cf3db1a4d4dcdc2f86c05f2e82afa310d0ecbb0379f2" exitCode=0 Oct 30 13:11:51.788999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.772217 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" event={"ID":"04fb46b3-a555-41ab-a7be-f57ed8c9deaf","Type":"ContainerDied","Data":"132f90033fd2e255d137cf3db1a4d4dcdc2f86c05f2e82afa310d0ecbb0379f2"} Oct 30 13:11:51.788999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.772297 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" event={"ID":"04fb46b3-a555-41ab-a7be-f57ed8c9deaf","Type":"ContainerDied","Data":"5ff1eda75abc662105d27ab75182dfe90a592a628437c8ebd1378e4ffebb63f6"} Oct 30 13:11:51.788999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.772334 4503 scope.go:122] "RemoveContainer" containerID="132f90033fd2e255d137cf3db1a4d4dcdc2f86c05f2e82afa310d0ecbb0379f2" Oct 30 13:11:51.799434 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.799179 4503 scope.go:122] "RemoveContainer" containerID="f024dbffe0f78e624a039c3ffe55080f2d5035d2d6b6b54088feeed6d5d78a96" Oct 30 13:11:51.809427 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.808602 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-kube-api-access-7r29x\" (UniqueName: \"kubernetes.io/projected/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-kube-api-access-7r29x\") pod \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " Oct 30 13:11:51.809427 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.808693 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-podinfo\") pod \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " Oct 30 13:11:51.809427 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.808796 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-credential-socket\") pod \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " Oct 30 13:11:51.809427 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.808834 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-workload-certs\") pod \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " Oct 30 13:11:51.809427 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.808878 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-workload-socket\") pod \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " Oct 30 13:11:51.809427 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.808910 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-data\") pod \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " Oct 30 13:11:51.809427 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.809093 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istiod-ca-cert\") pod \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " Oct 30 13:11:51.809427 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.809200 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-token\" (UniqueName: \"kubernetes.io/projected/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-token\") pod \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " Oct 30 13:11:51.809427 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.809287 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-envoy\") pod \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\" (UID: \"04fb46b3-a555-41ab-a7be-f57ed8c9deaf\") " Oct 30 13:11:51.812304 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.812203 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-credential-socket" pod "04fb46b3-a555-41ab-a7be-f57ed8c9deaf" (UID: "04fb46b3-a555-41ab-a7be-f57ed8c9deaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:51.812521 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.812452 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-data" pod "04fb46b3-a555-41ab-a7be-f57ed8c9deaf" (UID: "04fb46b3-a555-41ab-a7be-f57ed8c9deaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:51.812652 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.812632 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-workload-socket" pod "04fb46b3-a555-41ab-a7be-f57ed8c9deaf" (UID: "04fb46b3-a555-41ab-a7be-f57ed8c9deaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:51.813055 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.812906 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istiod-ca-cert" pod "04fb46b3-a555-41ab-a7be-f57ed8c9deaf" (UID: "04fb46b3-a555-41ab-a7be-f57ed8c9deaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:51.813418 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.813392 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-workload-certs" pod "04fb46b3-a555-41ab-a7be-f57ed8c9deaf" (UID: "04fb46b3-a555-41ab-a7be-f57ed8c9deaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:51.813559 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.813538 4503 scope.go:122] "RemoveContainer" containerID="132f90033fd2e255d137cf3db1a4d4dcdc2f86c05f2e82afa310d0ecbb0379f2" Oct 30 13:11:51.842221 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.841754 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-podinfo" pod "04fb46b3-a555-41ab-a7be-f57ed8c9deaf" (UID: "04fb46b3-a555-41ab-a7be-f57ed8c9deaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:51.842317 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.842215 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-envoy" pod "04fb46b3-a555-41ab-a7be-f57ed8c9deaf" (UID: "04fb46b3-a555-41ab-a7be-f57ed8c9deaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:51.842839 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.842795 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-kube-api-access-7r29x" pod "04fb46b3-a555-41ab-a7be-f57ed8c9deaf" (UID: "04fb46b3-a555-41ab-a7be-f57ed8c9deaf"). InnerVolumeSpecName "kube-api-access-7r29x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:51.843360 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:11:51.843321 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132f90033fd2e255d137cf3db1a4d4dcdc2f86c05f2e82afa310d0ecbb0379f2\": container with ID starting with 132f90033fd2e255d137cf3db1a4d4dcdc2f86c05f2e82afa310d0ecbb0379f2 not found: ID does not exist" containerID="132f90033fd2e255d137cf3db1a4d4dcdc2f86c05f2e82afa310d0ecbb0379f2" Oct 30 13:11:51.843439 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.843375 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132f90033fd2e255d137cf3db1a4d4dcdc2f86c05f2e82afa310d0ecbb0379f2"} err="failed to get container status \"132f90033fd2e255d137cf3db1a4d4dcdc2f86c05f2e82afa310d0ecbb0379f2\": rpc error: code = NotFound desc = could not find container \"132f90033fd2e255d137cf3db1a4d4dcdc2f86c05f2e82afa310d0ecbb0379f2\": container with ID starting with 132f90033fd2e255d137cf3db1a4d4dcdc2f86c05f2e82afa310d0ecbb0379f2 not found: ID does not exist" Oct 30 13:11:51.843439 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.843404 4503 scope.go:122] "RemoveContainer" containerID="f024dbffe0f78e624a039c3ffe55080f2d5035d2d6b6b54088feeed6d5d78a96" Oct 30 13:11:51.848344 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:11:51.846348 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f024dbffe0f78e624a039c3ffe55080f2d5035d2d6b6b54088feeed6d5d78a96\": container with ID starting with f024dbffe0f78e624a039c3ffe55080f2d5035d2d6b6b54088feeed6d5d78a96 not found: ID does not exist" containerID="f024dbffe0f78e624a039c3ffe55080f2d5035d2d6b6b54088feeed6d5d78a96" Oct 30 13:11:51.848344 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.846387 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f024dbffe0f78e624a039c3ffe55080f2d5035d2d6b6b54088feeed6d5d78a96"} err="failed to get container status \"f024dbffe0f78e624a039c3ffe55080f2d5035d2d6b6b54088feeed6d5d78a96\": rpc error: code = NotFound desc = could not find container \"f024dbffe0f78e624a039c3ffe55080f2d5035d2d6b6b54088feeed6d5d78a96\": container with ID starting with f024dbffe0f78e624a039c3ffe55080f2d5035d2d6b6b54088feeed6d5d78a96 not found: ID does not exist" Oct 30 13:11:51.848344 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.846614 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-token" pod "04fb46b3-a555-41ab-a7be-f57ed8c9deaf" (UID: "04fb46b3-a555-41ab-a7be-f57ed8c9deaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:51.911197 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.911164 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:51.911197 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.911196 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:51.911197 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.911207 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r29x\" (UniqueName: \"kubernetes.io/projected/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-kube-api-access-7r29x\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:51.911436 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.911216 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:51.911436 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.911226 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:51.911436 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.911234 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:51.911436 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.911243 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:51.911436 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.911251 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:51.911436 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:51.911260 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fb46b3-a555-41ab-a7be-f57ed8c9deaf-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:52.010161 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:52.010124 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9"] Oct 30 13:11:52.010795 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:11:52.010763 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692ff35a_d45a_4a6e_80f6_92511e76e7fd.slice/crio-22ed4ee65cc0f5cf937148fa932c4c1d886eb3e73bd99b50396a4efabcdec9f8 WatchSource:0}: Error finding container 22ed4ee65cc0f5cf937148fa932c4c1d886eb3e73bd99b50396a4efabcdec9f8: Status 404 returned error can't find the container with id 22ed4ee65cc0f5cf937148fa932c4c1d886eb3e73bd99b50396a4efabcdec9f8 Oct 30 13:11:52.014149 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:52.014105 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:52.014243 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:52.014207 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:52.014279 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:52.014255 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:11:52.786566 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:52.786535 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm" Oct 30 13:11:52.788615 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:52.788584 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" event={"ID":"692ff35a-d45a-4a6e-80f6-92511e76e7fd","Type":"ContainerStarted","Data":"b85fde405feeba3a867487bb397205d004ed700fe909419232ceadb662358960"} Oct 30 13:11:52.788789 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:52.788635 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" event={"ID":"692ff35a-d45a-4a6e-80f6-92511e76e7fd","Type":"ContainerStarted","Data":"22ed4ee65cc0f5cf937148fa932c4c1d886eb3e73bd99b50396a4efabcdec9f8"} Oct 30 13:11:52.881143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:52.881096 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm"] Oct 30 13:11:52.884767 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:52.884736 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-0fd5fcba-gw-openshift-default-695566bbcc-2wszm"] Oct 30 13:11:52.930905 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:52.930849 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" podStartSLOduration=2.930836432 podStartE2EDuration="2.930836432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:52.929187727 +0000 UTC m=+1206.752416006" watchObservedRunningTime="2026-10-30 13:11:52.930836432 +0000 UTC m=+1206.754064706" Oct 30 13:11:53.217773 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:53.217734 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:11:53.219595 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:53.219565 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 13:11:53.219726 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:53.219619 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" podUID="692ff35a-d45a-4a6e-80f6-92511e76e7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 13:11:54.217872 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:54.217833 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 13:11:54.218320 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:54.217895 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" podUID="692ff35a-d45a-4a6e-80f6-92511e76e7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 13:11:54.378104 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:54.378069 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04fb46b3-a555-41ab-a7be-f57ed8c9deaf" path="/var/lib/kubelet/pods/04fb46b3-a555-41ab-a7be-f57ed8c9deaf/volumes" Oct 30 13:11:55.218039 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:55.218002 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 13:11:55.218428 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:55.218056 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" podUID="692ff35a-d45a-4a6e-80f6-92511e76e7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 13:11:56.217046 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:56.217010 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 13:11:56.217256 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:56.217069 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" podUID="692ff35a-d45a-4a6e-80f6-92511e76e7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 13:11:56.830572 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:56.830330 4503 generic.go:356] "Generic (PLEG): container finished" podID="25ebe375-c4db-41eb-bc87-3cb9d8e0853b" containerID="3db6d80a5e252a56b71a9a0666c47917154ee121516f7df5b9e48d6449aa798a" exitCode=0 Oct 30 13:11:56.830572 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:56.830500 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" event={"ID":"25ebe375-c4db-41eb-bc87-3cb9d8e0853b","Type":"ContainerDied","Data":"3db6d80a5e252a56b71a9a0666c47917154ee121516f7df5b9e48d6449aa798a"} Oct 30 13:11:56.895191 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:56.836292 4503 generic.go:356] "Generic (PLEG): container finished" podID="81a74b8e-b1f2-467b-bb2c-6b0029e464f4" containerID="4af7d550e92bda682187fd81e5821e501d48dec4dd0f744701ee746d1e403f05" exitCode=0 Oct 30 13:11:56.895191 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:56.836383 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" event={"ID":"81a74b8e-b1f2-467b-bb2c-6b0029e464f4","Type":"ContainerDied","Data":"4af7d550e92bda682187fd81e5821e501d48dec4dd0f744701ee746d1e403f05"} Oct 30 13:11:57.092617 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.092583 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:11:57.118200 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.117563 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:11:57.129620 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.129591 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istiod-ca-cert\") pod \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " Oct 30 13:11:57.129787 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.129686 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-credential-socket\") pod \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " Oct 30 13:11:57.129787 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.129723 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-workload-certs\") pod \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " Oct 30 13:11:57.129787 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.129763 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-data\") pod \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " Oct 30 13:11:57.129983 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.129872 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-token\" (UniqueName: \"kubernetes.io/projected/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-token\") pod \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " Oct 30 13:11:57.129983 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.129897 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-podinfo\") pod \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " Oct 30 13:11:57.129983 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.129915 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-kube-api-access-rngnt\" (UniqueName: \"kubernetes.io/projected/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-kube-api-access-rngnt\") pod \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " Oct 30 13:11:57.129983 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.129932 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-envoy\") pod \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " Oct 30 13:11:57.130252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.129991 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-workload-socket\") pod \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\" (UID: \"81a74b8e-b1f2-467b-bb2c-6b0029e464f4\") " Oct 30 13:11:57.131404 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.130555 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-workload-socket" pod "81a74b8e-b1f2-467b-bb2c-6b0029e464f4" (UID: "81a74b8e-b1f2-467b-bb2c-6b0029e464f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:57.131404 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.130829 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istiod-ca-cert" pod "81a74b8e-b1f2-467b-bb2c-6b0029e464f4" (UID: "81a74b8e-b1f2-467b-bb2c-6b0029e464f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:57.131404 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.131016 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-credential-socket" pod "81a74b8e-b1f2-467b-bb2c-6b0029e464f4" (UID: "81a74b8e-b1f2-467b-bb2c-6b0029e464f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:57.131404 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.131131 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-workload-certs" pod "81a74b8e-b1f2-467b-bb2c-6b0029e464f4" (UID: "81a74b8e-b1f2-467b-bb2c-6b0029e464f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:57.131404 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.131237 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-data" pod "81a74b8e-b1f2-467b-bb2c-6b0029e464f4" (UID: "81a74b8e-b1f2-467b-bb2c-6b0029e464f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:57.143484 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.143457 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-podinfo" pod "81a74b8e-b1f2-467b-bb2c-6b0029e464f4" (UID: "81a74b8e-b1f2-467b-bb2c-6b0029e464f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:57.143648 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.143625 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-kube-api-access-rngnt" pod "81a74b8e-b1f2-467b-bb2c-6b0029e464f4" (UID: "81a74b8e-b1f2-467b-bb2c-6b0029e464f4"). InnerVolumeSpecName "kube-api-access-rngnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:57.143699 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.143506 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-token" pod "81a74b8e-b1f2-467b-bb2c-6b0029e464f4" (UID: "81a74b8e-b1f2-467b-bb2c-6b0029e464f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:57.191017 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.190694 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-envoy" pod "81a74b8e-b1f2-467b-bb2c-6b0029e464f4" (UID: "81a74b8e-b1f2-467b-bb2c-6b0029e464f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:57.217920 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.217883 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 13:11:57.218212 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.218182 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" podUID="692ff35a-d45a-4a6e-80f6-92511e76e7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 13:11:57.231366 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.231339 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-kube-api-access-dl84v\" (UniqueName: \"kubernetes.io/projected/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-kube-api-access-dl84v\") pod \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " Oct 30 13:11:57.231930 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.231911 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istiod-ca-cert\") pod \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " Oct 30 13:11:57.232118 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.232098 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-credential-socket\") pod \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " Oct 30 13:11:57.232322 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.232303 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-workload-socket\") pod \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " Oct 30 13:11:57.232430 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.232410 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istiod-ca-cert" pod "25ebe375-c4db-41eb-bc87-3cb9d8e0853b" (UID: "25ebe375-c4db-41eb-bc87-3cb9d8e0853b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:57.232572 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.232544 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-token\" (UniqueName: \"kubernetes.io/projected/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-token\") pod \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " Oct 30 13:11:57.232688 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.232668 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-credential-socket" pod "25ebe375-c4db-41eb-bc87-3cb9d8e0853b" (UID: "25ebe375-c4db-41eb-bc87-3cb9d8e0853b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:57.232865 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.232848 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-workload-certs\") pod \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " Oct 30 13:11:57.233200 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.232848 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-workload-socket" pod "25ebe375-c4db-41eb-bc87-3cb9d8e0853b" (UID: "25ebe375-c4db-41eb-bc87-3cb9d8e0853b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:57.233705 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.233438 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-workload-certs" pod "25ebe375-c4db-41eb-bc87-3cb9d8e0853b" (UID: "25ebe375-c4db-41eb-bc87-3cb9d8e0853b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:57.233828 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.233640 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-data" pod "25ebe375-c4db-41eb-bc87-3cb9d8e0853b" (UID: "25ebe375-c4db-41eb-bc87-3cb9d8e0853b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:57.233962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.233686 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-data\") pod \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " Oct 30 13:11:57.234165 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.234144 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-podinfo\") pod \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " Oct 30 13:11:57.234533 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.234516 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-envoy\") pod \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\" (UID: \"25ebe375-c4db-41eb-bc87-3cb9d8e0853b\") " Oct 30 13:11:57.235590 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.235563 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.235804 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.235790 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.235911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.235898 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.236069 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.236056 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.236166 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.236155 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.236302 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.236291 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.236441 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.236429 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.236542 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.236531 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.236708 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.236692 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rngnt\" (UniqueName: \"kubernetes.io/projected/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-kube-api-access-rngnt\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.236827 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.236814 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.237041 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.236928 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.237162 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.237147 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.237271 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.237258 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.237388 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.237372 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a74b8e-b1f2-467b-bb2c-6b0029e464f4-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.237488 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.236108 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-token" pod "25ebe375-c4db-41eb-bc87-3cb9d8e0853b" (UID: "25ebe375-c4db-41eb-bc87-3cb9d8e0853b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:57.237582 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.236597 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-kube-api-access-dl84v" pod "25ebe375-c4db-41eb-bc87-3cb9d8e0853b" (UID: "25ebe375-c4db-41eb-bc87-3cb9d8e0853b"). InnerVolumeSpecName "kube-api-access-dl84v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:57.237794 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.237774 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-podinfo" pod "25ebe375-c4db-41eb-bc87-3cb9d8e0853b" (UID: "25ebe375-c4db-41eb-bc87-3cb9d8e0853b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:57.238618 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.238588 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-envoy" pod "25ebe375-c4db-41eb-bc87-3cb9d8e0853b" (UID: "25ebe375-c4db-41eb-bc87-3cb9d8e0853b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:57.293700 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.293667 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:11:57.338650 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.338614 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788f38f0-c8b1-461b-969e-8d171ad33149-kube-api-access-wtbzg\" (UniqueName: \"kubernetes.io/projected/788f38f0-c8b1-461b-969e-8d171ad33149-kube-api-access-wtbzg\") pod \"788f38f0-c8b1-461b-969e-8d171ad33149\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " Oct 30 13:11:57.338844 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.338695 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/788f38f0-c8b1-461b-969e-8d171ad33149-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788f38f0-c8b1-461b-969e-8d171ad33149-istio-podinfo\") pod \"788f38f0-c8b1-461b-969e-8d171ad33149\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " Oct 30 13:11:57.338844 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.338722 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-istio-data\") pod \"788f38f0-c8b1-461b-969e-8d171ad33149\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " Oct 30 13:11:57.338844 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.338828 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/788f38f0-c8b1-461b-969e-8d171ad33149-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788f38f0-c8b1-461b-969e-8d171ad33149-istiod-ca-cert\") pod \"788f38f0-c8b1-461b-969e-8d171ad33149\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " Oct 30 13:11:57.339047 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.338887 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-workload-certs\") pod \"788f38f0-c8b1-461b-969e-8d171ad33149\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " Oct 30 13:11:57.339047 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.339010 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-istio-envoy\") pod \"788f38f0-c8b1-461b-969e-8d171ad33149\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " Oct 30 13:11:57.339215 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.339191 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788f38f0-c8b1-461b-969e-8d171ad33149-istio-token\" (UniqueName: \"kubernetes.io/projected/788f38f0-c8b1-461b-969e-8d171ad33149-istio-token\") pod \"788f38f0-c8b1-461b-969e-8d171ad33149\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " Oct 30 13:11:57.339299 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.339260 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-workload-socket\") pod \"788f38f0-c8b1-461b-969e-8d171ad33149\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " Oct 30 13:11:57.339299 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.339271 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-istio-data" pod "788f38f0-c8b1-461b-969e-8d171ad33149" (UID: "788f38f0-c8b1-461b-969e-8d171ad33149"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:57.339423 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.339315 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-credential-socket\") pod \"788f38f0-c8b1-461b-969e-8d171ad33149\" (UID: \"788f38f0-c8b1-461b-969e-8d171ad33149\") " Oct 30 13:11:57.339423 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.339328 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-workload-certs" pod "788f38f0-c8b1-461b-969e-8d171ad33149" (UID: "788f38f0-c8b1-461b-969e-8d171ad33149"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:57.339529 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.339413 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788f38f0-c8b1-461b-969e-8d171ad33149-istiod-ca-cert" pod "788f38f0-c8b1-461b-969e-8d171ad33149" (UID: "788f38f0-c8b1-461b-969e-8d171ad33149"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:57.339595 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.339556 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-workload-socket" pod "788f38f0-c8b1-461b-969e-8d171ad33149" (UID: "788f38f0-c8b1-461b-969e-8d171ad33149"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:57.339778 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.339751 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-credential-socket" pod "788f38f0-c8b1-461b-969e-8d171ad33149" (UID: "788f38f0-c8b1-461b-969e-8d171ad33149"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:57.340070 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.340046 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.340181 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.340074 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788f38f0-c8b1-461b-969e-8d171ad33149-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.340181 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.340089 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.340181 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.340102 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.340181 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.340115 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.340181 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.340127 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl84v\" (UniqueName: \"kubernetes.io/projected/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-kube-api-access-dl84v\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.340181 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.340152 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.340181 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.340167 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25ebe375-c4db-41eb-bc87-3cb9d8e0853b-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.340181 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.340180 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.341349 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.341319 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/788f38f0-c8b1-461b-969e-8d171ad33149-istio-podinfo" pod "788f38f0-c8b1-461b-969e-8d171ad33149" (UID: "788f38f0-c8b1-461b-969e-8d171ad33149"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:57.341461 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.341386 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788f38f0-c8b1-461b-969e-8d171ad33149-kube-api-access-wtbzg" pod "788f38f0-c8b1-461b-969e-8d171ad33149" (UID: "788f38f0-c8b1-461b-969e-8d171ad33149"). InnerVolumeSpecName "kube-api-access-wtbzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:57.341461 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.341430 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-istio-envoy" pod "788f38f0-c8b1-461b-969e-8d171ad33149" (UID: "788f38f0-c8b1-461b-969e-8d171ad33149"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:57.341559 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.341465 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788f38f0-c8b1-461b-969e-8d171ad33149-istio-token" pod "788f38f0-c8b1-461b-969e-8d171ad33149" (UID: "788f38f0-c8b1-461b-969e-8d171ad33149"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:57.441735 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.441639 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788f38f0-c8b1-461b-969e-8d171ad33149-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.441735 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.441675 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtbzg\" (UniqueName: \"kubernetes.io/projected/788f38f0-c8b1-461b-969e-8d171ad33149-kube-api-access-wtbzg\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.441735 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.441688 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788f38f0-c8b1-461b-969e-8d171ad33149-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.441735 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.441701 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788f38f0-c8b1-461b-969e-8d171ad33149-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:11:57.872590 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.861656 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" event={"ID":"25ebe375-c4db-41eb-bc87-3cb9d8e0853b","Type":"ContainerDied","Data":"a022bab8d1b3991aa9d9ce49e201a57b8f4394906ce651045a2abbaa8a77e0f9"} Oct 30 13:11:57.872590 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.861740 4503 scope.go:122] "RemoveContainer" containerID="3db6d80a5e252a56b71a9a0666c47917154ee121516f7df5b9e48d6449aa798a" Oct 30 13:11:57.872590 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.861915 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6" Oct 30 13:11:57.882991 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.881529 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" Oct 30 13:11:57.882991 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.881526 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp" event={"ID":"81a74b8e-b1f2-467b-bb2c-6b0029e464f4","Type":"ContainerDied","Data":"4685e9707abc716fc8f2444eda968f643152cf71764a3cf19b4d672288843a40"} Oct 30 13:11:57.884322 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.884293 4503 generic.go:356] "Generic (PLEG): container finished" podID="788f38f0-c8b1-461b-969e-8d171ad33149" containerID="b4663661b422579d5b7c09addfc9808b6aad52711328bc82dc422abd2b326947" exitCode=0 Oct 30 13:11:57.884450 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.884396 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" event={"ID":"788f38f0-c8b1-461b-969e-8d171ad33149","Type":"ContainerDied","Data":"b4663661b422579d5b7c09addfc9808b6aad52711328bc82dc422abd2b326947"} Oct 30 13:11:57.884450 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.884436 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" event={"ID":"788f38f0-c8b1-461b-969e-8d171ad33149","Type":"ContainerDied","Data":"974c12b0d757d6cfcd3bf8212265dca79509979f64cf3f513c946270255dbe1a"} Oct 30 13:11:57.884532 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.884506 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m" Oct 30 13:11:57.891618 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.891518 4503 scope.go:122] "RemoveContainer" containerID="4af7d550e92bda682187fd81e5821e501d48dec4dd0f744701ee746d1e403f05" Oct 30 13:11:57.905712 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.905692 4503 scope.go:122] "RemoveContainer" containerID="b4663661b422579d5b7c09addfc9808b6aad52711328bc82dc422abd2b326947" Oct 30 13:11:57.915668 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.915647 4503 scope.go:122] "RemoveContainer" containerID="b4663661b422579d5b7c09addfc9808b6aad52711328bc82dc422abd2b326947" Oct 30 13:11:57.915980 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:11:57.915955 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4663661b422579d5b7c09addfc9808b6aad52711328bc82dc422abd2b326947\": container with ID starting with b4663661b422579d5b7c09addfc9808b6aad52711328bc82dc422abd2b326947 not found: ID does not exist" containerID="b4663661b422579d5b7c09addfc9808b6aad52711328bc82dc422abd2b326947" Oct 30 13:11:57.916064 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.915990 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4663661b422579d5b7c09addfc9808b6aad52711328bc82dc422abd2b326947"} err="failed to get container status \"b4663661b422579d5b7c09addfc9808b6aad52711328bc82dc422abd2b326947\": rpc error: code = NotFound desc = could not find container \"b4663661b422579d5b7c09addfc9808b6aad52711328bc82dc422abd2b326947\": container with ID starting with b4663661b422579d5b7c09addfc9808b6aad52711328bc82dc422abd2b326947 not found: ID does not exist" Oct 30 13:11:57.989242 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:57.988089 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6"] Oct 30 13:11:58.111179 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:58.111137 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-g9tf6"] Oct 30 13:11:58.187046 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:58.183293 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp"] Oct 30 13:11:58.188367 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:58.188341 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-xxxzp"] Oct 30 13:11:58.236536 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:58.236494 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m"] Oct 30 13:11:58.237457 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:58.237427 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 13:11:58.237694 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:58.237667 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" podUID="692ff35a-d45a-4a6e-80f6-92511e76e7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 13:11:58.242975 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:58.242889 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3048f9-openshift-default-89cb9455f-8mp4m"] Oct 30 13:11:58.375962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:58.375077 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25ebe375-c4db-41eb-bc87-3cb9d8e0853b" path="/var/lib/kubelet/pods/25ebe375-c4db-41eb-bc87-3cb9d8e0853b/volumes" Oct 30 13:11:58.375962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:58.375841 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="788f38f0-c8b1-461b-969e-8d171ad33149" path="/var/lib/kubelet/pods/788f38f0-c8b1-461b-969e-8d171ad33149/volumes" Oct 30 13:11:58.386979 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:58.376482 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a74b8e-b1f2-467b-bb2c-6b0029e464f4" path="/var/lib/kubelet/pods/81a74b8e-b1f2-467b-bb2c-6b0029e464f4/volumes" Oct 30 13:11:59.217461 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:59.217381 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 13:11:59.217461 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:11:59.217435 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" podUID="692ff35a-d45a-4a6e-80f6-92511e76e7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 13:11:59.710676 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:11:59.710633 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:12:00.221482 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:00.221448 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:12:00.221952 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:00.221913 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:12:00.222633 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:00.222616 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:12:20.635988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:20.628028 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 13:12:20.653962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:20.652012 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:12:20.916966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:20.905116 4503 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:12:21.027283 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:21.027086 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" podUID="9361bd06-84ab-404c-8893-e61490265ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:12:21.902359 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:21.902320 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h"] Oct 30 13:12:21.902773 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:21.902620 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" podUID="c1423ef5-6c20-40a5-a870-4934b313ba4c" containerName="istio-proxy" containerID="cri-o://eb30591e2d53cb3587c35832487f18850ea2664b090f887e370a3fa29d3527db" gracePeriod=30 Oct 30 13:12:21.913022 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:21.912994 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f"] Oct 30 13:12:21.913202 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:21.913182 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" podUID="c715c318-f465-4b5a-8c65-10c6539a5c61" containerName="istio-proxy" containerID="cri-o://504c3f6fb85b53ecbe3eebb10ab3a6c972b6645431801a4e4c12671c503ea418" gracePeriod=30 Oct 30 13:12:21.918248 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:21.918219 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d"] Oct 30 13:12:21.918466 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:21.918436 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" podUID="c1d0fc5e-a51f-490d-892a-809a3464c0b6" containerName="istio-proxy" containerID="cri-o://2c5cd52c2f885a5902ec0a75c2449ae37a2f8bafc649f5c60585f9ee3c1c923d" gracePeriod=30 Oct 30 13:12:21.923552 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:21.923522 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw"] Oct 30 13:12:21.923707 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:21.923689 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" podUID="72c4c258-7da6-40a5-82f5-618f8bea345f" containerName="istio-proxy" containerID="cri-o://8f2a566a128b7ac0b05681b0d74edb405d97067f4bc81d9dd0172351fd4cbacd" gracePeriod=30 Oct 30 13:12:23.282791 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:23.282745 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh"] Oct 30 13:12:26.488570 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:26.488535 4503 patch_prober.go:29] interesting pod/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:12:26.489000 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:26.488591 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" podUID="c1423ef5-6c20-40a5-a870-4934b313ba4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:12:27.027606 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:12:27.027567 4503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c4c258_7da6_40a5_82f5_618f8bea345f.slice/crio-8f2a566a128b7ac0b05681b0d74edb405d97067f4bc81d9dd0172351fd4cbacd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c4c258_7da6_40a5_82f5_618f8bea345f.slice/crio-conmon-8f2a566a128b7ac0b05681b0d74edb405d97067f4bc81d9dd0172351fd4cbacd.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:12:27.281247 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.281212 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:12:27.283125 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.283096 4503 generic.go:356] "Generic (PLEG): container finished" podID="c1423ef5-6c20-40a5-a870-4934b313ba4c" containerID="eb30591e2d53cb3587c35832487f18850ea2664b090f887e370a3fa29d3527db" exitCode=0 Oct 30 13:12:27.283263 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.283246 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" event={"ID":"c1423ef5-6c20-40a5-a870-4934b313ba4c","Type":"ContainerDied","Data":"eb30591e2d53cb3587c35832487f18850ea2664b090f887e370a3fa29d3527db"} Oct 30 13:12:27.285417 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.285219 4503 generic.go:356] "Generic (PLEG): container finished" podID="c715c318-f465-4b5a-8c65-10c6539a5c61" containerID="504c3f6fb85b53ecbe3eebb10ab3a6c972b6645431801a4e4c12671c503ea418" exitCode=0 Oct 30 13:12:27.285417 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.285351 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" event={"ID":"c715c318-f465-4b5a-8c65-10c6539a5c61","Type":"ContainerDied","Data":"504c3f6fb85b53ecbe3eebb10ab3a6c972b6645431801a4e4c12671c503ea418"} Oct 30 13:12:27.289385 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.288336 4503 generic.go:356] "Generic (PLEG): container finished" podID="72c4c258-7da6-40a5-82f5-618f8bea345f" containerID="8f2a566a128b7ac0b05681b0d74edb405d97067f4bc81d9dd0172351fd4cbacd" exitCode=0 Oct 30 13:12:27.289385 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.288463 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" event={"ID":"72c4c258-7da6-40a5-82f5-618f8bea345f","Type":"ContainerDied","Data":"8f2a566a128b7ac0b05681b0d74edb405d97067f4bc81d9dd0172351fd4cbacd"} Oct 30 13:12:27.290218 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.290155 4503 generic.go:356] "Generic (PLEG): container finished" podID="c1d0fc5e-a51f-490d-892a-809a3464c0b6" containerID="2c5cd52c2f885a5902ec0a75c2449ae37a2f8bafc649f5c60585f9ee3c1c923d" exitCode=0 Oct 30 13:12:27.290343 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.290253 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" event={"ID":"c1d0fc5e-a51f-490d-892a-809a3464c0b6","Type":"ContainerDied","Data":"2c5cd52c2f885a5902ec0a75c2449ae37a2f8bafc649f5c60585f9ee3c1c923d"} Oct 30 13:12:27.290343 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.290294 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" event={"ID":"c1d0fc5e-a51f-490d-892a-809a3464c0b6","Type":"ContainerDied","Data":"3c4dba5be3f220b2ba3bd77026d88c8f610a4bc8569e7f7adfc02b259424aed2"} Oct 30 13:12:27.290343 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.290321 4503 scope.go:122] "RemoveContainer" containerID="2c5cd52c2f885a5902ec0a75c2449ae37a2f8bafc649f5c60585f9ee3c1c923d" Oct 30 13:12:27.290515 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.290481 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d" Oct 30 13:12:27.304707 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.304676 4503 scope.go:122] "RemoveContainer" containerID="2c5cd52c2f885a5902ec0a75c2449ae37a2f8bafc649f5c60585f9ee3c1c923d" Oct 30 13:12:27.305093 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:12:27.305011 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c5cd52c2f885a5902ec0a75c2449ae37a2f8bafc649f5c60585f9ee3c1c923d\": container with ID starting with 2c5cd52c2f885a5902ec0a75c2449ae37a2f8bafc649f5c60585f9ee3c1c923d not found: ID does not exist" containerID="2c5cd52c2f885a5902ec0a75c2449ae37a2f8bafc649f5c60585f9ee3c1c923d" Oct 30 13:12:27.305093 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.305048 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5cd52c2f885a5902ec0a75c2449ae37a2f8bafc649f5c60585f9ee3c1c923d"} err="failed to get container status \"2c5cd52c2f885a5902ec0a75c2449ae37a2f8bafc649f5c60585f9ee3c1c923d\": rpc error: code = NotFound desc = could not find container \"2c5cd52c2f885a5902ec0a75c2449ae37a2f8bafc649f5c60585f9ee3c1c923d\": container with ID starting with 2c5cd52c2f885a5902ec0a75c2449ae37a2f8bafc649f5c60585f9ee3c1c923d not found: ID does not exist" Oct 30 13:12:27.405325 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.405289 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d0fc5e-a51f-490d-892a-809a3464c0b6-kube-api-access-z7wtv\" (UniqueName: \"kubernetes.io/projected/c1d0fc5e-a51f-490d-892a-809a3464c0b6-kube-api-access-z7wtv\") pod \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " Oct 30 13:12:27.405465 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.405398 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-workload-certs\") pod \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " Oct 30 13:12:27.405465 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.405452 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-data\") pod \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " Oct 30 13:12:27.405590 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.405515 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-credential-socket\") pod \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " Oct 30 13:12:27.405645 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.405608 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-envoy\") pod \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " Oct 30 13:12:27.405691 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.405638 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-podinfo\") pod \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " Oct 30 13:12:27.405749 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.405734 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-workload-socket\") pod \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " Oct 30 13:12:27.405852 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.405835 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-token\" (UniqueName: \"kubernetes.io/projected/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-token\") pod \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " Oct 30 13:12:27.405927 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.405912 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istiod-ca-cert\") pod \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\" (UID: \"c1d0fc5e-a51f-490d-892a-809a3464c0b6\") " Oct 30 13:12:27.407364 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.407338 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istiod-ca-cert" pod "c1d0fc5e-a51f-490d-892a-809a3464c0b6" (UID: "c1d0fc5e-a51f-490d-892a-809a3464c0b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:27.407613 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.407509 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-workload-certs" pod "c1d0fc5e-a51f-490d-892a-809a3464c0b6" (UID: "c1d0fc5e-a51f-490d-892a-809a3464c0b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.407707 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.407520 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-data" pod "c1d0fc5e-a51f-490d-892a-809a3464c0b6" (UID: "c1d0fc5e-a51f-490d-892a-809a3464c0b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.408109 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.408084 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-credential-socket" pod "c1d0fc5e-a51f-490d-892a-809a3464c0b6" (UID: "c1d0fc5e-a51f-490d-892a-809a3464c0b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.408322 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.408306 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-workload-socket" pod "c1d0fc5e-a51f-490d-892a-809a3464c0b6" (UID: "c1d0fc5e-a51f-490d-892a-809a3464c0b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.411269 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.411251 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-token" pod "c1d0fc5e-a51f-490d-892a-809a3464c0b6" (UID: "c1d0fc5e-a51f-490d-892a-809a3464c0b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:27.411615 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.411554 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d0fc5e-a51f-490d-892a-809a3464c0b6-kube-api-access-z7wtv" pod "c1d0fc5e-a51f-490d-892a-809a3464c0b6" (UID: "c1d0fc5e-a51f-490d-892a-809a3464c0b6"). InnerVolumeSpecName "kube-api-access-z7wtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:27.412515 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.412498 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-envoy" pod "c1d0fc5e-a51f-490d-892a-809a3464c0b6" (UID: "c1d0fc5e-a51f-490d-892a-809a3464c0b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.413510 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.413472 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-podinfo" pod "c1d0fc5e-a51f-490d-892a-809a3464c0b6" (UID: "c1d0fc5e-a51f-490d-892a-809a3464c0b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:12:27.417329 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.416862 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:12:27.420226 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.420042 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:12:27.423459 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.423440 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:12:27.508147 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.508057 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-workload-certs\") pod \"c1423ef5-6c20-40a5-a870-4934b313ba4c\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " Oct 30 13:12:27.508147 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.508112 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1423ef5-6c20-40a5-a870-4934b313ba4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1423ef5-6c20-40a5-a870-4934b313ba4c-istiod-ca-cert\") pod \"c1423ef5-6c20-40a5-a870-4934b313ba4c\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " Oct 30 13:12:27.508673 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.508147 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72c4c258-7da6-40a5-82f5-618f8bea345f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c4c258-7da6-40a5-82f5-618f8bea345f-istiod-ca-cert\") pod \"72c4c258-7da6-40a5-82f5-618f8bea345f\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " Oct 30 13:12:27.508673 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.508184 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-data\") pod \"c1423ef5-6c20-40a5-a870-4934b313ba4c\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " Oct 30 13:12:27.508673 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.508358 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-podinfo\") pod \"72c4c258-7da6-40a5-82f5-618f8bea345f\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " Oct 30 13:12:27.508673 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.508417 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-workload-certs" pod "c1423ef5-6c20-40a5-a870-4934b313ba4c" (UID: "c1423ef5-6c20-40a5-a870-4934b313ba4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.508673 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.508497 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-data" pod "c1423ef5-6c20-40a5-a870-4934b313ba4c" (UID: "c1423ef5-6c20-40a5-a870-4934b313ba4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.508673 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.508577 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c4c258-7da6-40a5-82f5-618f8bea345f-istiod-ca-cert" pod "72c4c258-7da6-40a5-82f5-618f8bea345f" (UID: "72c4c258-7da6-40a5-82f5-618f8bea345f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:27.508673 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.508640 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1423ef5-6c20-40a5-a870-4934b313ba4c-istiod-ca-cert" pod "c1423ef5-6c20-40a5-a870-4934b313ba4c" (UID: "c1423ef5-6c20-40a5-a870-4934b313ba4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:27.508901 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.508883 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-credential-socket\") pod \"72c4c258-7da6-40a5-82f5-618f8bea345f\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " Oct 30 13:12:27.508953 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.508922 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-data\") pod \"72c4c258-7da6-40a5-82f5-618f8bea345f\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " Oct 30 13:12:27.509051 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509030 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1423ef5-6c20-40a5-a870-4934b313ba4c-kube-api-access-x5kzh\" (UniqueName: \"kubernetes.io/projected/c1423ef5-6c20-40a5-a870-4934b313ba4c-kube-api-access-x5kzh\") pod \"c1423ef5-6c20-40a5-a870-4934b313ba4c\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " Oct 30 13:12:27.509130 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509105 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-workload-socket\") pod \"c715c318-f465-4b5a-8c65-10c6539a5c61\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " Oct 30 13:12:27.509313 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509241 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c4c258-7da6-40a5-82f5-618f8bea345f-kube-api-access-hhzxt\" (UniqueName: \"kubernetes.io/projected/72c4c258-7da6-40a5-82f5-618f8bea345f-kube-api-access-hhzxt\") pod \"72c4c258-7da6-40a5-82f5-618f8bea345f\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " Oct 30 13:12:27.509313 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509281 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-credential-socket" pod "72c4c258-7da6-40a5-82f5-618f8bea345f" (UID: "72c4c258-7da6-40a5-82f5-618f8bea345f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.509313 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509290 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c715c318-f465-4b5a-8c65-10c6539a5c61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c715c318-f465-4b5a-8c65-10c6539a5c61-istiod-ca-cert\") pod \"c715c318-f465-4b5a-8c65-10c6539a5c61\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " Oct 30 13:12:27.509313 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509298 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-data" pod "72c4c258-7da6-40a5-82f5-618f8bea345f" (UID: "72c4c258-7da6-40a5-82f5-618f8bea345f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.509570 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509322 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-token\" (UniqueName: \"kubernetes.io/projected/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-token\") pod \"c715c318-f465-4b5a-8c65-10c6539a5c61\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " Oct 30 13:12:27.509570 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509369 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-workload-certs\") pod \"c715c318-f465-4b5a-8c65-10c6539a5c61\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " Oct 30 13:12:27.509570 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509378 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-workload-socket" pod "c715c318-f465-4b5a-8c65-10c6539a5c61" (UID: "c715c318-f465-4b5a-8c65-10c6539a5c61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.509570 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509406 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-workload-socket\") pod \"c1423ef5-6c20-40a5-a870-4934b313ba4c\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " Oct 30 13:12:27.509570 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509451 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c715c318-f465-4b5a-8c65-10c6539a5c61-kube-api-access-zbqlr\" (UniqueName: \"kubernetes.io/projected/c715c318-f465-4b5a-8c65-10c6539a5c61-kube-api-access-zbqlr\") pod \"c715c318-f465-4b5a-8c65-10c6539a5c61\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " Oct 30 13:12:27.509570 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509494 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-data\") pod \"c715c318-f465-4b5a-8c65-10c6539a5c61\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " Oct 30 13:12:27.509570 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509532 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-credential-socket\") pod \"c1423ef5-6c20-40a5-a870-4934b313ba4c\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " Oct 30 13:12:27.509570 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509567 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-envoy\") pod \"72c4c258-7da6-40a5-82f5-618f8bea345f\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " Oct 30 13:12:27.510021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509619 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-credential-socket\") pod \"c715c318-f465-4b5a-8c65-10c6539a5c61\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " Oct 30 13:12:27.510021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509649 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c715c318-f465-4b5a-8c65-10c6539a5c61-istiod-ca-cert" pod "c715c318-f465-4b5a-8c65-10c6539a5c61" (UID: "c715c318-f465-4b5a-8c65-10c6539a5c61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:27.510021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509661 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-workload-socket\") pod \"72c4c258-7da6-40a5-82f5-618f8bea345f\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " Oct 30 13:12:27.510021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509694 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-workload-certs\") pod \"72c4c258-7da6-40a5-82f5-618f8bea345f\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " Oct 30 13:12:27.510021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509746 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-podinfo\") pod \"c1423ef5-6c20-40a5-a870-4934b313ba4c\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " Oct 30 13:12:27.510021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509817 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-envoy\") pod \"c715c318-f465-4b5a-8c65-10c6539a5c61\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " Oct 30 13:12:27.510021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509862 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-podinfo\") pod \"c715c318-f465-4b5a-8c65-10c6539a5c61\" (UID: \"c715c318-f465-4b5a-8c65-10c6539a5c61\") " Oct 30 13:12:27.510021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509900 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-token\" (UniqueName: \"kubernetes.io/projected/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-token\") pod \"c1423ef5-6c20-40a5-a870-4934b313ba4c\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " Oct 30 13:12:27.510021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509928 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-credential-socket" pod "c715c318-f465-4b5a-8c65-10c6539a5c61" (UID: "c715c318-f465-4b5a-8c65-10c6539a5c61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.510021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.509959 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-token\" (UniqueName: \"kubernetes.io/projected/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-token\") pod \"72c4c258-7da6-40a5-82f5-618f8bea345f\" (UID: \"72c4c258-7da6-40a5-82f5-618f8bea345f\") " Oct 30 13:12:27.510021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.510000 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-envoy\") pod \"c1423ef5-6c20-40a5-a870-4934b313ba4c\" (UID: \"c1423ef5-6c20-40a5-a870-4934b313ba4c\") " Oct 30 13:12:27.510839 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.510420 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-workload-socket" pod "c1423ef5-6c20-40a5-a870-4934b313ba4c" (UID: "c1423ef5-6c20-40a5-a870-4934b313ba4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.510839 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.510794 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.510839 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.510819 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.510839 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.510819 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-data" pod "c715c318-f465-4b5a-8c65-10c6539a5c61" (UID: "c715c318-f465-4b5a-8c65-10c6539a5c61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.510839 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.510835 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.511150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.510850 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.511150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.510865 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.511150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.510879 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.511150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.510891 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.511150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.510903 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1423ef5-6c20-40a5-a870-4934b313ba4c-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.511150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.510916 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c4c258-7da6-40a5-82f5-618f8bea345f-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.511150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.510929 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.511150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.510960 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7wtv\" (UniqueName: \"kubernetes.io/projected/c1d0fc5e-a51f-490d-892a-809a3464c0b6-kube-api-access-z7wtv\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.511150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.510973 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.511150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.510987 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.511150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.511000 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.511150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.511012 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.511150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.511025 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.511150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.511037 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c715c318-f465-4b5a-8c65-10c6539a5c61-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.511150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.511051 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d0fc5e-a51f-490d-892a-809a3464c0b6-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.511150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.511064 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.511150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.511122 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-workload-certs" pod "c715c318-f465-4b5a-8c65-10c6539a5c61" (UID: "c715c318-f465-4b5a-8c65-10c6539a5c61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.512045 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.511282 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-workload-certs" pod "72c4c258-7da6-40a5-82f5-618f8bea345f" (UID: "72c4c258-7da6-40a5-82f5-618f8bea345f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.512045 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.511482 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-workload-socket" pod "72c4c258-7da6-40a5-82f5-618f8bea345f" (UID: "72c4c258-7da6-40a5-82f5-618f8bea345f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.512525 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.512488 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-credential-socket" pod "c1423ef5-6c20-40a5-a870-4934b313ba4c" (UID: "c1423ef5-6c20-40a5-a870-4934b313ba4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.512714 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.512692 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-token" pod "c715c318-f465-4b5a-8c65-10c6539a5c61" (UID: "c715c318-f465-4b5a-8c65-10c6539a5c61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:27.512882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.512855 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-envoy" pod "c1423ef5-6c20-40a5-a870-4934b313ba4c" (UID: "c1423ef5-6c20-40a5-a870-4934b313ba4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.512980 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.512928 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-podinfo" pod "72c4c258-7da6-40a5-82f5-618f8bea345f" (UID: "72c4c258-7da6-40a5-82f5-618f8bea345f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:12:27.513832 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.513796 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-podinfo" pod "c1423ef5-6c20-40a5-a870-4934b313ba4c" (UID: "c1423ef5-6c20-40a5-a870-4934b313ba4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:12:27.514322 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.514285 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c4c258-7da6-40a5-82f5-618f8bea345f-kube-api-access-hhzxt" pod "72c4c258-7da6-40a5-82f5-618f8bea345f" (UID: "72c4c258-7da6-40a5-82f5-618f8bea345f"). InnerVolumeSpecName "kube-api-access-hhzxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:27.520859 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.514660 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-envoy" pod "c715c318-f465-4b5a-8c65-10c6539a5c61" (UID: "c715c318-f465-4b5a-8c65-10c6539a5c61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.520859 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.514688 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1423ef5-6c20-40a5-a870-4934b313ba4c-kube-api-access-x5kzh" pod "c1423ef5-6c20-40a5-a870-4934b313ba4c" (UID: "c1423ef5-6c20-40a5-a870-4934b313ba4c"). InnerVolumeSpecName "kube-api-access-x5kzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:27.520859 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.517102 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-envoy" pod "72c4c258-7da6-40a5-82f5-618f8bea345f" (UID: "72c4c258-7da6-40a5-82f5-618f8bea345f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:27.520859 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.517118 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-token" pod "72c4c258-7da6-40a5-82f5-618f8bea345f" (UID: "72c4c258-7da6-40a5-82f5-618f8bea345f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:27.520859 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.517150 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c715c318-f465-4b5a-8c65-10c6539a5c61-kube-api-access-zbqlr" pod "c715c318-f465-4b5a-8c65-10c6539a5c61" (UID: "c715c318-f465-4b5a-8c65-10c6539a5c61"). InnerVolumeSpecName "kube-api-access-zbqlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:27.520859 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.517172 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-podinfo" pod "c715c318-f465-4b5a-8c65-10c6539a5c61" (UID: "c715c318-f465-4b5a-8c65-10c6539a5c61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:12:27.520859 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.518050 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-token" pod "c1423ef5-6c20-40a5-a870-4934b313ba4c" (UID: "c1423ef5-6c20-40a5-a870-4934b313ba4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:27.612533 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.612501 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.612533 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.612530 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5kzh\" (UniqueName: \"kubernetes.io/projected/c1423ef5-6c20-40a5-a870-4934b313ba4c-kube-api-access-x5kzh\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.612533 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.612541 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhzxt\" (UniqueName: \"kubernetes.io/projected/72c4c258-7da6-40a5-82f5-618f8bea345f-kube-api-access-hhzxt\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.612802 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.612551 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.612802 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.612561 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.612802 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.612569 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbqlr\" (UniqueName: \"kubernetes.io/projected/c715c318-f465-4b5a-8c65-10c6539a5c61-kube-api-access-zbqlr\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.612802 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.612578 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.612802 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.612586 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.612802 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.612594 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.612802 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.612603 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.612802 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.612612 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c4c258-7da6-40a5-82f5-618f8bea345f-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.612802 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.612620 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.612802 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.612628 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.612802 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.612637 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c715c318-f465-4b5a-8c65-10c6539a5c61-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.612802 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.612645 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.612802 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.612653 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c4c258-7da6-40a5-82f5-618f8bea345f-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.612802 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.612661 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1423ef5-6c20-40a5-a870-4934b313ba4c-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:27.648113 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.648075 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d"] Oct 30 13:12:27.655377 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:27.655344 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-nbn4d"] Oct 30 13:12:28.302263 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:28.302180 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" Oct 30 13:12:28.302263 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:28.302179 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h" event={"ID":"c1423ef5-6c20-40a5-a870-4934b313ba4c","Type":"ContainerDied","Data":"ee581d67aa7fb40f5d49f4db6548d810cd6595a80272e8c2817f47a64508e21a"} Oct 30 13:12:28.302494 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:28.302331 4503 scope.go:122] "RemoveContainer" containerID="eb30591e2d53cb3587c35832487f18850ea2664b090f887e370a3fa29d3527db" Oct 30 13:12:28.303634 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:28.303617 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" Oct 30 13:12:28.303634 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:28.303619 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f" event={"ID":"c715c318-f465-4b5a-8c65-10c6539a5c61","Type":"ContainerDied","Data":"ee684c4cecc846e2d3028567117242ae6cb1cf0de37d9d23d0dcd862f1a161f7"} Oct 30 13:12:28.310356 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:28.310316 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" event={"ID":"72c4c258-7da6-40a5-82f5-618f8bea345f","Type":"ContainerDied","Data":"21ca8323d66d6b836a62957b3ec847a781a978835137c43ea376baca4b6e984a"} Oct 30 13:12:28.310471 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:28.310331 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw" Oct 30 13:12:28.314205 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:28.314173 4503 scope.go:122] "RemoveContainer" containerID="504c3f6fb85b53ecbe3eebb10ab3a6c972b6645431801a4e4c12671c503ea418" Oct 30 13:12:28.325238 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:28.325220 4503 scope.go:122] "RemoveContainer" containerID="8f2a566a128b7ac0b05681b0d74edb405d97067f4bc81d9dd0172351fd4cbacd" Oct 30 13:12:28.363014 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:28.362983 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d0fc5e-a51f-490d-892a-809a3464c0b6" path="/var/lib/kubelet/pods/c1d0fc5e-a51f-490d-892a-809a3464c0b6/volumes" Oct 30 13:12:28.372462 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:28.372429 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h"] Oct 30 13:12:28.384398 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:28.384371 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-6fn8h"] Oct 30 13:12:28.414806 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:28.414781 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f"] Oct 30 13:12:28.421925 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:28.421898 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-2r24f"] Oct 30 13:12:28.459903 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:28.459876 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw"] Oct 30 13:12:28.465053 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:28.465029 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-484607-openshift-default-b8b957995-4lrdw"] Oct 30 13:12:29.038633 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:29.038596 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw"] Oct 30 13:12:29.798965 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:12:29.794668 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:12:30.362807 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:30.362775 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72c4c258-7da6-40a5-82f5-618f8bea345f" path="/var/lib/kubelet/pods/72c4c258-7da6-40a5-82f5-618f8bea345f/volumes" Oct 30 13:12:30.363368 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:30.363337 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1423ef5-6c20-40a5-a870-4934b313ba4c" path="/var/lib/kubelet/pods/c1423ef5-6c20-40a5-a870-4934b313ba4c/volumes" Oct 30 13:12:30.363975 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:30.363954 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c715c318-f465-4b5a-8c65-10c6539a5c61" path="/var/lib/kubelet/pods/c715c318-f465-4b5a-8c65-10c6539a5c61/volumes" Oct 30 13:12:34.038915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:34.038867 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw" podUID="738b8f50-96a6-4a49-b5f7-92b1b9fb5975" containerName="payload-pre-processing" containerID="cri-o://7015fa22b21a65f0cea24e6026ae78af2c4e197d1074f5db4119e17307818502" gracePeriod=25 Oct 30 13:12:34.337590 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:34.337563 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw" Oct 30 13:12:34.375620 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:34.375586 4503 generic.go:356] "Generic (PLEG): container finished" podID="738b8f50-96a6-4a49-b5f7-92b1b9fb5975" containerID="7015fa22b21a65f0cea24e6026ae78af2c4e197d1074f5db4119e17307818502" exitCode=0 Oct 30 13:12:34.375782 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:34.375629 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw" Oct 30 13:12:34.375782 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:34.375661 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw" event={"ID":"738b8f50-96a6-4a49-b5f7-92b1b9fb5975","Type":"ContainerDied","Data":"7015fa22b21a65f0cea24e6026ae78af2c4e197d1074f5db4119e17307818502"} Oct 30 13:12:34.375782 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:34.375719 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw" event={"ID":"738b8f50-96a6-4a49-b5f7-92b1b9fb5975","Type":"ContainerDied","Data":"a27a73680138853938cc3ba617cc8eb923c95b5ef99677f2385326b3ec61d93a"} Oct 30 13:12:34.375782 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:34.375746 4503 scope.go:122] "RemoveContainer" containerID="7015fa22b21a65f0cea24e6026ae78af2c4e197d1074f5db4119e17307818502" Oct 30 13:12:34.388825 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:34.388800 4503 scope.go:122] "RemoveContainer" containerID="7015fa22b21a65f0cea24e6026ae78af2c4e197d1074f5db4119e17307818502" Oct 30 13:12:34.389214 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:12:34.389186 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7015fa22b21a65f0cea24e6026ae78af2c4e197d1074f5db4119e17307818502\": container with ID starting with 7015fa22b21a65f0cea24e6026ae78af2c4e197d1074f5db4119e17307818502 not found: ID does not exist" containerID="7015fa22b21a65f0cea24e6026ae78af2c4e197d1074f5db4119e17307818502" Oct 30 13:12:34.389364 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:34.389218 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7015fa22b21a65f0cea24e6026ae78af2c4e197d1074f5db4119e17307818502"} err="failed to get container status \"7015fa22b21a65f0cea24e6026ae78af2c4e197d1074f5db4119e17307818502\": rpc error: code = NotFound desc = could not find container \"7015fa22b21a65f0cea24e6026ae78af2c4e197d1074f5db4119e17307818502\": container with ID starting with 7015fa22b21a65f0cea24e6026ae78af2c4e197d1074f5db4119e17307818502 not found: ID does not exist" Oct 30 13:12:34.445296 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:34.445263 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/738b8f50-96a6-4a49-b5f7-92b1b9fb5975-kube-api-access-7sr4j\" (UniqueName: \"kubernetes.io/projected/738b8f50-96a6-4a49-b5f7-92b1b9fb5975-kube-api-access-7sr4j\") pod \"738b8f50-96a6-4a49-b5f7-92b1b9fb5975\" (UID: \"738b8f50-96a6-4a49-b5f7-92b1b9fb5975\") " Oct 30 13:12:34.445544 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:34.445527 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/738b8f50-96a6-4a49-b5f7-92b1b9fb5975-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/738b8f50-96a6-4a49-b5f7-92b1b9fb5975-plugins-config-volume\") pod \"738b8f50-96a6-4a49-b5f7-92b1b9fb5975\" (UID: \"738b8f50-96a6-4a49-b5f7-92b1b9fb5975\") " Oct 30 13:12:34.445923 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:34.445895 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/738b8f50-96a6-4a49-b5f7-92b1b9fb5975-plugins-config-volume" pod "738b8f50-96a6-4a49-b5f7-92b1b9fb5975" (UID: "738b8f50-96a6-4a49-b5f7-92b1b9fb5975"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:34.447709 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:34.447679 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/738b8f50-96a6-4a49-b5f7-92b1b9fb5975-kube-api-access-7sr4j" pod "738b8f50-96a6-4a49-b5f7-92b1b9fb5975" (UID: "738b8f50-96a6-4a49-b5f7-92b1b9fb5975"). InnerVolumeSpecName "kube-api-access-7sr4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:34.546969 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:34.546923 4503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/738b8f50-96a6-4a49-b5f7-92b1b9fb5975-plugins-config-volume\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:34.546969 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:34.546970 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sr4j\" (UniqueName: \"kubernetes.io/projected/738b8f50-96a6-4a49-b5f7-92b1b9fb5975-kube-api-access-7sr4j\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:34.725714 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:34.725671 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw"] Oct 30 13:12:34.735672 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:34.735641 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ac0c58-8b8558d87-qslrw"] Oct 30 13:12:35.729838 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.729804 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb"] Oct 30 13:12:35.730380 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730365 4503 cpu_manager.go:414] "containerMap: removing container" podUID="70f0f890-bb6a-4627-9487-90741d34337c" containerName="istio-proxy" Oct 30 13:12:35.730424 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730381 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f0f890-bb6a-4627-9487-90741d34337c" containerName="istio-proxy" Oct 30 13:12:35.730424 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730396 4503 cpu_manager.go:414] "containerMap: removing container" podUID="c715c318-f465-4b5a-8c65-10c6539a5c61" containerName="istio-proxy" Oct 30 13:12:35.730424 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730401 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c715c318-f465-4b5a-8c65-10c6539a5c61" containerName="istio-proxy" Oct 30 13:12:35.730424 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730411 4503 cpu_manager.go:414] "containerMap: removing container" podUID="04fb46b3-a555-41ab-a7be-f57ed8c9deaf" containerName="istio-proxy" Oct 30 13:12:35.730424 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730415 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04fb46b3-a555-41ab-a7be-f57ed8c9deaf" containerName="istio-proxy" Oct 30 13:12:35.730580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730434 4503 cpu_manager.go:414] "containerMap: removing container" podUID="25ebe375-c4db-41eb-bc87-3cb9d8e0853b" containerName="istio-proxy" Oct 30 13:12:35.730580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730438 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ebe375-c4db-41eb-bc87-3cb9d8e0853b" containerName="istio-proxy" Oct 30 13:12:35.730580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730450 4503 cpu_manager.go:414] "containerMap: removing container" podUID="c1423ef5-6c20-40a5-a870-4934b313ba4c" containerName="istio-proxy" Oct 30 13:12:35.730580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730455 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1423ef5-6c20-40a5-a870-4934b313ba4c" containerName="istio-proxy" Oct 30 13:12:35.730580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730463 4503 cpu_manager.go:414] "containerMap: removing container" podUID="738b8f50-96a6-4a49-b5f7-92b1b9fb5975" containerName="payload-pre-processing" Oct 30 13:12:35.730580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730468 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="738b8f50-96a6-4a49-b5f7-92b1b9fb5975" containerName="payload-pre-processing" Oct 30 13:12:35.730580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730477 4503 cpu_manager.go:414] "containerMap: removing container" podUID="c1d0fc5e-a51f-490d-892a-809a3464c0b6" containerName="istio-proxy" Oct 30 13:12:35.730580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730481 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d0fc5e-a51f-490d-892a-809a3464c0b6" containerName="istio-proxy" Oct 30 13:12:35.730580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730496 4503 cpu_manager.go:414] "containerMap: removing container" podUID="72c4c258-7da6-40a5-82f5-618f8bea345f" containerName="istio-proxy" Oct 30 13:12:35.730580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730501 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c4c258-7da6-40a5-82f5-618f8bea345f" containerName="istio-proxy" Oct 30 13:12:35.730580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730509 4503 cpu_manager.go:414] "containerMap: removing container" podUID="81a74b8e-b1f2-467b-bb2c-6b0029e464f4" containerName="istio-proxy" Oct 30 13:12:35.730580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730513 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a74b8e-b1f2-467b-bb2c-6b0029e464f4" containerName="istio-proxy" Oct 30 13:12:35.730580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730523 4503 cpu_manager.go:414] "containerMap: removing container" podUID="788f38f0-c8b1-461b-969e-8d171ad33149" containerName="istio-proxy" Oct 30 13:12:35.730580 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730527 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="788f38f0-c8b1-461b-969e-8d171ad33149" containerName="istio-proxy" Oct 30 13:12:35.731002 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730603 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ebe375-c4db-41eb-bc87-3cb9d8e0853b" containerName="istio-proxy" Oct 30 13:12:35.731002 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730612 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1423ef5-6c20-40a5-a870-4934b313ba4c" containerName="istio-proxy" Oct 30 13:12:35.731002 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730619 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="738b8f50-96a6-4a49-b5f7-92b1b9fb5975" containerName="payload-pre-processing" Oct 30 13:12:35.731002 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730624 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d0fc5e-a51f-490d-892a-809a3464c0b6" containerName="istio-proxy" Oct 30 13:12:35.731002 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730632 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c4c258-7da6-40a5-82f5-618f8bea345f" containerName="istio-proxy" Oct 30 13:12:35.731002 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730637 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a74b8e-b1f2-467b-bb2c-6b0029e464f4" containerName="istio-proxy" Oct 30 13:12:35.731002 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730643 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="788f38f0-c8b1-461b-969e-8d171ad33149" containerName="istio-proxy" Oct 30 13:12:35.731002 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730649 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="04fb46b3-a555-41ab-a7be-f57ed8c9deaf" containerName="istio-proxy" Oct 30 13:12:35.731002 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730655 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="c715c318-f465-4b5a-8c65-10c6539a5c61" containerName="istio-proxy" Oct 30 13:12:35.731002 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.730660 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="04fb46b3-a555-41ab-a7be-f57ed8c9deaf" containerName="istio-proxy" Oct 30 13:12:35.733880 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.733860 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.762508 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.762473 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb"] Oct 30 13:12:35.867251 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.867207 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istiod-ca-cert\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.867432 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.867337 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-podinfo\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.867432 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.867392 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9tm2\" (UniqueName: \"kubernetes.io/projected/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-kube-api-access-p9tm2\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.867432 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.867419 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-workload-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.867657 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.867632 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-workload-certs\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.867698 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.867686 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-data\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.867821 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.867804 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-credential-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.867862 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.867845 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-token\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.867966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.867951 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-envoy\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.969201 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.969164 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9tm2\" (UniqueName: \"kubernetes.io/projected/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-kube-api-access-p9tm2\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.969396 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.969208 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-workload-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.969396 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.969293 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-workload-certs\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.969396 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.969331 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-data\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.969573 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.969423 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-credential-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.969573 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.969446 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-token\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.969573 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.969531 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-envoy\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.969573 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.969554 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istiod-ca-cert\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.969791 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.969603 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-podinfo\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.969849 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.969787 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-workload-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.969997 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.969824 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-workload-certs\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.969997 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.969989 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-data\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.970481 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.970456 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-credential-socket\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.970877 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.970851 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istiod-ca-cert\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.974439 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.973128 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-envoy\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.982791 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.982728 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-podinfo\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.994147 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.994120 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-token\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:35.994228 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:35.994183 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9tm2\" (UniqueName: \"kubernetes.io/projected/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-kube-api-access-p9tm2\") pod \"e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:36.050494 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:36.050460 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:36.363301 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:36.363222 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="738b8f50-96a6-4a49-b5f7-92b1b9fb5975" path="/var/lib/kubelet/pods/738b8f50-96a6-4a49-b5f7-92b1b9fb5975/volumes" Oct 30 13:12:36.377421 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:36.377394 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb"] Oct 30 13:12:36.378665 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:12:36.378635 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8afe3e_7c5d_4ba1_93a2_898433af86d7.slice/crio-e88b0c3ebaa9302ca789aea7a2de6d05578b0640b37c4ee7870cff999805927f WatchSource:0}: Error finding container e88b0c3ebaa9302ca789aea7a2de6d05578b0640b37c4ee7870cff999805927f: Status 404 returned error can't find the container with id e88b0c3ebaa9302ca789aea7a2de6d05578b0640b37c4ee7870cff999805927f Oct 30 13:12:36.380877 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:36.380817 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:36.380984 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:36.380920 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:36.381059 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:36.380980 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:36.390999 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:36.390972 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" event={"ID":"1a8afe3e-7c5d-4ba1-93a2-898433af86d7","Type":"ContainerStarted","Data":"e88b0c3ebaa9302ca789aea7a2de6d05578b0640b37c4ee7870cff999805927f"} Oct 30 13:12:37.400797 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:37.400759 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" event={"ID":"1a8afe3e-7c5d-4ba1-93a2-898433af86d7","Type":"ContainerStarted","Data":"120b4cc973950e7b36dc5995c281476a8bdd7bfdffa40f10d7c8459974fcf6d0"} Oct 30 13:12:37.480361 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:37.480299 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" podStartSLOduration=2.4802826319999998 podStartE2EDuration="2.480282632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:37.46739044 +0000 UTC m=+1251.290618718" watchObservedRunningTime="2026-10-30 13:12:37.480282632 +0000 UTC m=+1251.303510909" Oct 30 13:12:38.067092 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:38.061597 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:38.074524 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:38.070085 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 13:12:38.074524 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:38.070214 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" podUID="1a8afe3e-7c5d-4ba1-93a2-898433af86d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 13:12:38.310171 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:38.310126 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" podUID="9361bd06-84ab-404c-8893-e61490265ce2" containerName="main" containerID="cri-o://556c17b2d4cc9257106c677da1cae0896e831fa6daaffd51bf7c6ef18d9ede3c" gracePeriod=45 Oct 30 13:12:38.603787 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:38.603686 4503 generic.go:356] "Generic (PLEG): container finished" podID="9361bd06-84ab-404c-8893-e61490265ce2" containerID="556c17b2d4cc9257106c677da1cae0896e831fa6daaffd51bf7c6ef18d9ede3c" exitCode=0 Oct 30 13:12:38.605358 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:38.605323 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" event={"ID":"9361bd06-84ab-404c-8893-e61490265ce2","Type":"ContainerDied","Data":"556c17b2d4cc9257106c677da1cae0896e831fa6daaffd51bf7c6ef18d9ede3c"} Oct 30 13:12:38.843741 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:38.843718 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:12:39.017176 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.012629 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-tmp-dir\") pod \"9361bd06-84ab-404c-8893-e61490265ce2\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " Oct 30 13:12:39.017176 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.012726 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-model-cache\") pod \"9361bd06-84ab-404c-8893-e61490265ce2\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " Oct 30 13:12:39.017176 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.012827 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9361bd06-84ab-404c-8893-e61490265ce2-tls-certs\" (UniqueName: \"kubernetes.io/secret/9361bd06-84ab-404c-8893-e61490265ce2-tls-certs\") pod \"9361bd06-84ab-404c-8893-e61490265ce2\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " Oct 30 13:12:39.017176 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.012862 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-home\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-home\") pod \"9361bd06-84ab-404c-8893-e61490265ce2\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " Oct 30 13:12:39.017176 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.012998 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9361bd06-84ab-404c-8893-e61490265ce2-kube-api-access-r8brd\" (UniqueName: \"kubernetes.io/projected/9361bd06-84ab-404c-8893-e61490265ce2-kube-api-access-r8brd\") pod \"9361bd06-84ab-404c-8893-e61490265ce2\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " Oct 30 13:12:39.017176 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.013091 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-dshm\") pod \"9361bd06-84ab-404c-8893-e61490265ce2\" (UID: \"9361bd06-84ab-404c-8893-e61490265ce2\") " Oct 30 13:12:39.017176 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.015499 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-tmp-dir" pod "9361bd06-84ab-404c-8893-e61490265ce2" (UID: "9361bd06-84ab-404c-8893-e61490265ce2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:39.017176 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.015757 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-model-cache" pod "9361bd06-84ab-404c-8893-e61490265ce2" (UID: "9361bd06-84ab-404c-8893-e61490265ce2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:39.017176 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.016339 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-home" pod "9361bd06-84ab-404c-8893-e61490265ce2" (UID: "9361bd06-84ab-404c-8893-e61490265ce2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:39.017788 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.017692 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-dshm" pod "9361bd06-84ab-404c-8893-e61490265ce2" (UID: "9361bd06-84ab-404c-8893-e61490265ce2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:39.018722 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.018693 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9361bd06-84ab-404c-8893-e61490265ce2-tls-certs" pod "9361bd06-84ab-404c-8893-e61490265ce2" (UID: "9361bd06-84ab-404c-8893-e61490265ce2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:39.031310 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.031273 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9361bd06-84ab-404c-8893-e61490265ce2-kube-api-access-r8brd" pod "9361bd06-84ab-404c-8893-e61490265ce2" (UID: "9361bd06-84ab-404c-8893-e61490265ce2"). InnerVolumeSpecName "kube-api-access-r8brd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:39.070986 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.068311 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 13:12:39.070986 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.068379 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" podUID="1a8afe3e-7c5d-4ba1-93a2-898433af86d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 13:12:39.115565 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.115520 4503 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9361bd06-84ab-404c-8893-e61490265ce2-tls-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:39.115565 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.115560 4503 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-home\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:39.115565 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.115571 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8brd\" (UniqueName: \"kubernetes.io/projected/9361bd06-84ab-404c-8893-e61490265ce2-kube-api-access-r8brd\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:39.115824 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.115580 4503 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-dshm\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:39.115824 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.115590 4503 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-tmp-dir\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:39.115824 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.115599 4503 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9361bd06-84ab-404c-8893-e61490265ce2-model-cache\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:39.612031 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.612000 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" Oct 30 13:12:39.612484 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.612050 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh" event={"ID":"9361bd06-84ab-404c-8893-e61490265ce2","Type":"ContainerDied","Data":"c9b775575f1a262c12f9642ec077f0bd186899e2ab76e646d11fdbd8b0c22b8f"} Oct 30 13:12:39.612484 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.612098 4503 scope.go:122] "RemoveContainer" containerID="556c17b2d4cc9257106c677da1cae0896e831fa6daaffd51bf7c6ef18d9ede3c" Oct 30 13:12:39.675821 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.675777 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh"] Oct 30 13:12:39.691916 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:39.691877 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-ac0c58/distinct-2-w1-ac0c58-kserve-6ccb5c8d75-99xxh"] Oct 30 13:12:40.052404 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.052362 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 13:12:40.052701 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.052672 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" podUID="1a8afe3e-7c5d-4ba1-93a2-898433af86d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 13:12:40.278628 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.278585 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc"] Oct 30 13:12:40.279530 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.279493 4503 cpu_manager.go:414] "containerMap: removing container" podUID="9361bd06-84ab-404c-8893-e61490265ce2" containerName="main" Oct 30 13:12:40.279530 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.279532 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9361bd06-84ab-404c-8893-e61490265ce2" containerName="main" Oct 30 13:12:40.279658 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.279556 4503 cpu_manager.go:414] "containerMap: removing container" podUID="04fb46b3-a555-41ab-a7be-f57ed8c9deaf" containerName="istio-proxy" Oct 30 13:12:40.279658 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.279564 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04fb46b3-a555-41ab-a7be-f57ed8c9deaf" containerName="istio-proxy" Oct 30 13:12:40.279759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.279746 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="9361bd06-84ab-404c-8893-e61490265ce2" containerName="main" Oct 30 13:12:40.286927 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.286892 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc" Oct 30 13:12:40.308515 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.304208 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:12:40.310665 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.310427 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc"] Oct 30 13:12:40.316542 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.316507 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:12:40.391606 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.391576 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9361bd06-84ab-404c-8893-e61490265ce2" path="/var/lib/kubelet/pods/9361bd06-84ab-404c-8893-e61490265ce2/volumes" Oct 30 13:12:40.440305 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.440269 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57516645-0006-4f61-b699-dbc55090aecb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc\" (UID: \"57516645-0006-4f61-b699-dbc55090aecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc" Oct 30 13:12:40.440493 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.440393 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57516645-0006-4f61-b699-dbc55090aecb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc\" (UID: \"57516645-0006-4f61-b699-dbc55090aecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc" Oct 30 13:12:40.440493 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.440470 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqfqn\" (UniqueName: \"kubernetes.io/projected/57516645-0006-4f61-b699-dbc55090aecb-kube-api-access-mqfqn\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc\" (UID: \"57516645-0006-4f61-b699-dbc55090aecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc" Oct 30 13:12:40.541640 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.541600 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57516645-0006-4f61-b699-dbc55090aecb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc\" (UID: \"57516645-0006-4f61-b699-dbc55090aecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc" Oct 30 13:12:40.541856 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.541738 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57516645-0006-4f61-b699-dbc55090aecb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc\" (UID: \"57516645-0006-4f61-b699-dbc55090aecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc" Oct 30 13:12:40.542098 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.542063 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqfqn\" (UniqueName: \"kubernetes.io/projected/57516645-0006-4f61-b699-dbc55090aecb-kube-api-access-mqfqn\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc\" (UID: \"57516645-0006-4f61-b699-dbc55090aecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc" Oct 30 13:12:40.542378 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.542353 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57516645-0006-4f61-b699-dbc55090aecb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc\" (UID: \"57516645-0006-4f61-b699-dbc55090aecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc" Oct 30 13:12:40.545987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.544238 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57516645-0006-4f61-b699-dbc55090aecb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc\" (UID: \"57516645-0006-4f61-b699-dbc55090aecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc" Oct 30 13:12:40.573008 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.569641 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqfqn\" (UniqueName: \"kubernetes.io/projected/57516645-0006-4f61-b699-dbc55090aecb-kube-api-access-mqfqn\") pod \"maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc\" (UID: \"57516645-0006-4f61-b699-dbc55090aecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc" Oct 30 13:12:40.605387 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.605351 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc" Oct 30 13:12:40.979974 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:40.977033 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc"] Oct 30 13:12:40.985075 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:12:40.985046 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57516645_0006_4f61_b699_dbc55090aecb.slice/crio-93dbc373fac8eaf35dc0bf6509f163408c2731e137407c07de0f1eb95b33b2bd WatchSource:0}: Error finding container 93dbc373fac8eaf35dc0bf6509f163408c2731e137407c07de0f1eb95b33b2bd: Status 404 returned error can't find the container with id 93dbc373fac8eaf35dc0bf6509f163408c2731e137407c07de0f1eb95b33b2bd Oct 30 13:12:41.078984 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:41.072502 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 13:12:41.078984 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:41.072571 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" podUID="1a8afe3e-7c5d-4ba1-93a2-898433af86d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 13:12:41.630778 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:41.630744 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc" event={"ID":"57516645-0006-4f61-b699-dbc55090aecb","Type":"ContainerStarted","Data":"5ccf727a9eab881fef4dd8c31f67fd24a70f68d885a1ba8c25dcdb63cad186a1"} Oct 30 13:12:41.631059 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:41.630800 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc" event={"ID":"57516645-0006-4f61-b699-dbc55090aecb","Type":"ContainerStarted","Data":"93dbc373fac8eaf35dc0bf6509f163408c2731e137407c07de0f1eb95b33b2bd"} Oct 30 13:12:41.682529 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:41.682425 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc" podStartSLOduration=1.682408927 podStartE2EDuration="1.682408927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:41.681337658 +0000 UTC m=+1255.504565916" watchObservedRunningTime="2026-10-30 13:12:41.682408927 +0000 UTC m=+1255.505637203" Oct 30 13:12:42.075070 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:42.066324 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 13:12:42.075070 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:42.066390 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" podUID="1a8afe3e-7c5d-4ba1-93a2-898433af86d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 13:12:42.638683 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:42.638649 4503 generic.go:356] "Generic (PLEG): container finished" podID="57516645-0006-4f61-b699-dbc55090aecb" containerID="5ccf727a9eab881fef4dd8c31f67fd24a70f68d885a1ba8c25dcdb63cad186a1" exitCode=0 Oct 30 13:12:42.638922 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:42.638755 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc" event={"ID":"57516645-0006-4f61-b699-dbc55090aecb","Type":"ContainerDied","Data":"5ccf727a9eab881fef4dd8c31f67fd24a70f68d885a1ba8c25dcdb63cad186a1"} Oct 30 13:12:43.051767 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:43.051727 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 13:12:43.051947 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:43.051798 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" podUID="1a8afe3e-7c5d-4ba1-93a2-898433af86d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 13:12:43.834061 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:43.834031 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc" Oct 30 13:12:43.940770 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:43.940660 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57516645-0006-4f61-b699-dbc55090aecb-kube-api-access-mqfqn\" (UniqueName: \"kubernetes.io/projected/57516645-0006-4f61-b699-dbc55090aecb-kube-api-access-mqfqn\") pod \"57516645-0006-4f61-b699-dbc55090aecb\" (UID: \"57516645-0006-4f61-b699-dbc55090aecb\") " Oct 30 13:12:43.940954 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:43.940832 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57516645-0006-4f61-b699-dbc55090aecb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57516645-0006-4f61-b699-dbc55090aecb-maas-api-xks-ca\") pod \"57516645-0006-4f61-b699-dbc55090aecb\" (UID: \"57516645-0006-4f61-b699-dbc55090aecb\") " Oct 30 13:12:43.941025 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:43.941005 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57516645-0006-4f61-b699-dbc55090aecb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57516645-0006-4f61-b699-dbc55090aecb-maas-api-openshift-ca\") pod \"57516645-0006-4f61-b699-dbc55090aecb\" (UID: \"57516645-0006-4f61-b699-dbc55090aecb\") " Oct 30 13:12:43.941543 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:43.941512 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57516645-0006-4f61-b699-dbc55090aecb-maas-api-openshift-ca" pod "57516645-0006-4f61-b699-dbc55090aecb" (UID: "57516645-0006-4f61-b699-dbc55090aecb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:43.943410 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:43.943377 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57516645-0006-4f61-b699-dbc55090aecb-kube-api-access-mqfqn" pod "57516645-0006-4f61-b699-dbc55090aecb" (UID: "57516645-0006-4f61-b699-dbc55090aecb"). InnerVolumeSpecName "kube-api-access-mqfqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:43.943632 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:43.943606 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57516645-0006-4f61-b699-dbc55090aecb-maas-api-xks-ca" pod "57516645-0006-4f61-b699-dbc55090aecb" (UID: "57516645-0006-4f61-b699-dbc55090aecb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:44.043263 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:44.043176 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqfqn\" (UniqueName: \"kubernetes.io/projected/57516645-0006-4f61-b699-dbc55090aecb-kube-api-access-mqfqn\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:44.043263 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:44.043209 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57516645-0006-4f61-b699-dbc55090aecb-maas-api-xks-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:44.043263 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:44.043219 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57516645-0006-4f61-b699-dbc55090aecb-maas-api-openshift-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:44.056051 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:44.056014 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:44.056416 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:44.056387 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:44.057206 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:44.057182 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:12:44.657219 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:44.657189 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc" Oct 30 13:12:44.657417 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:44.657219 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc" event={"ID":"57516645-0006-4f61-b699-dbc55090aecb","Type":"ContainerDied","Data":"93dbc373fac8eaf35dc0bf6509f163408c2731e137407c07de0f1eb95b33b2bd"} Oct 30 13:12:44.657417 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:44.657270 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93dbc373fac8eaf35dc0bf6509f163408c2731e137407c07de0f1eb95b33b2bd" Oct 30 13:12:44.822002 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:44.821963 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf"] Oct 30 13:12:44.822290 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:44.822244 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" podUID="c3c12544-6e6c-487a-ad99-bb5d0f5da8aa" containerName="maas-api" containerID="cri-o://5831ec2f76cd31572b8db0c77700db7d44bc466fdecf2d7a4ba9985173459059" gracePeriod=30 Oct 30 13:12:45.226086 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.222480 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl"] Oct 30 13:12:45.238543 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.238474 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" Oct 30 13:12:45.241500 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.241473 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s"] Oct 30 13:12:45.321424 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.321387 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-openshift-service-ca\") pod \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\" (UID: \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\") " Oct 30 13:12:45.321606 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.321590 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-metrics-certs\") pod \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\" (UID: \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\") " Oct 30 13:12:45.321697 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.321682 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-kube-api-access-bjfkw\" (UniqueName: \"kubernetes.io/projected/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-kube-api-access-bjfkw\") pod \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\" (UID: \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\") " Oct 30 13:12:45.321814 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.321793 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-maas-api-tls\") pod \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\" (UID: \"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa\") " Oct 30 13:12:45.321918 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.321803 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-openshift-service-ca" pod "c3c12544-6e6c-487a-ad99-bb5d0f5da8aa" (UID: "c3c12544-6e6c-487a-ad99-bb5d0f5da8aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:45.322221 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.322190 4503 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-openshift-service-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:45.324408 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.324384 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-metrics-certs" pod "c3c12544-6e6c-487a-ad99-bb5d0f5da8aa" (UID: "c3c12544-6e6c-487a-ad99-bb5d0f5da8aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:45.324487 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.324410 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-kube-api-access-bjfkw" pod "c3c12544-6e6c-487a-ad99-bb5d0f5da8aa" (UID: "c3c12544-6e6c-487a-ad99-bb5d0f5da8aa"). InnerVolumeSpecName "kube-api-access-bjfkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:45.324665 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.324642 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-maas-api-tls" pod "c3c12544-6e6c-487a-ad99-bb5d0f5da8aa" (UID: "c3c12544-6e6c-487a-ad99-bb5d0f5da8aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:45.425975 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.424964 4503 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-metrics-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:45.425975 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.425009 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjfkw\" (UniqueName: \"kubernetes.io/projected/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-kube-api-access-bjfkw\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:45.425975 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.425029 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa-maas-api-tls\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:45.670479 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.670441 4503 generic.go:356] "Generic (PLEG): container finished" podID="c3c12544-6e6c-487a-ad99-bb5d0f5da8aa" containerID="5831ec2f76cd31572b8db0c77700db7d44bc466fdecf2d7a4ba9985173459059" exitCode=0 Oct 30 13:12:45.670644 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.670490 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" Oct 30 13:12:45.670644 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.670531 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" event={"ID":"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa","Type":"ContainerDied","Data":"5831ec2f76cd31572b8db0c77700db7d44bc466fdecf2d7a4ba9985173459059"} Oct 30 13:12:45.670644 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.670602 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf" event={"ID":"c3c12544-6e6c-487a-ad99-bb5d0f5da8aa","Type":"ContainerDied","Data":"836403b9ae41d49be179c10e1d513c52b98c9c5f48033ffee81ca144b31b73e6"} Oct 30 13:12:45.670644 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.670635 4503 scope.go:122] "RemoveContainer" containerID="5831ec2f76cd31572b8db0c77700db7d44bc466fdecf2d7a4ba9985173459059" Oct 30 13:12:45.683138 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.683113 4503 scope.go:122] "RemoveContainer" containerID="5831ec2f76cd31572b8db0c77700db7d44bc466fdecf2d7a4ba9985173459059" Oct 30 13:12:45.683476 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:12:45.683452 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5831ec2f76cd31572b8db0c77700db7d44bc466fdecf2d7a4ba9985173459059\": container with ID starting with 5831ec2f76cd31572b8db0c77700db7d44bc466fdecf2d7a4ba9985173459059 not found: ID does not exist" containerID="5831ec2f76cd31572b8db0c77700db7d44bc466fdecf2d7a4ba9985173459059" Oct 30 13:12:45.683573 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.683487 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5831ec2f76cd31572b8db0c77700db7d44bc466fdecf2d7a4ba9985173459059"} err="failed to get container status \"5831ec2f76cd31572b8db0c77700db7d44bc466fdecf2d7a4ba9985173459059\": rpc error: code = NotFound desc = could not find container \"5831ec2f76cd31572b8db0c77700db7d44bc466fdecf2d7a4ba9985173459059\": container with ID starting with 5831ec2f76cd31572b8db0c77700db7d44bc466fdecf2d7a4ba9985173459059 not found: ID does not exist" Oct 30 13:12:45.726653 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.726619 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf"] Oct 30 13:12:45.738244 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:45.738213 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ac0c58-6d499b7c69-8c5jf"] Oct 30 13:12:46.366219 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:46.366181 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c12544-6e6c-487a-ad99-bb5d0f5da8aa" path="/var/lib/kubelet/pods/c3c12544-6e6c-487a-ad99-bb5d0f5da8aa/volumes" Oct 30 13:12:50.766535 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:50.766493 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr"] Oct 30 13:12:52.073963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.070007 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w"] Oct 30 13:12:52.084529 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.083843 4503 cpu_manager.go:414] "containerMap: removing container" podUID="57516645-0006-4f61-b699-dbc55090aecb" containerName="revoke-keys" Oct 30 13:12:52.084529 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.083876 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57516645-0006-4f61-b699-dbc55090aecb" containerName="revoke-keys" Oct 30 13:12:52.084529 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.083930 4503 cpu_manager.go:414] "containerMap: removing container" podUID="c3c12544-6e6c-487a-ad99-bb5d0f5da8aa" containerName="maas-api" Oct 30 13:12:52.084529 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.083953 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c12544-6e6c-487a-ad99-bb5d0f5da8aa" containerName="maas-api" Oct 30 13:12:52.084529 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.084084 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="57516645-0006-4f61-b699-dbc55090aecb" containerName="revoke-keys" Oct 30 13:12:52.084529 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.084103 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c12544-6e6c-487a-ad99-bb5d0f5da8aa" containerName="maas-api" Oct 30 13:12:52.109964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.103767 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w" Oct 30 13:12:52.178471 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.178429 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwb5x\" (UniqueName: \"kubernetes.io/projected/08749638-af48-4615-9551-3b34e32233b3-kube-api-access-pwb5x\") pod \"maas-api-revoke-sub-345fe60c-mhg2w\" (UID: \"08749638-af48-4615-9551-3b34e32233b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w" Oct 30 13:12:52.186966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.178810 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08749638-af48-4615-9551-3b34e32233b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-345fe60c-mhg2w\" (UID: \"08749638-af48-4615-9551-3b34e32233b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w" Oct 30 13:12:52.196982 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.188350 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08749638-af48-4615-9551-3b34e32233b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-345fe60c-mhg2w\" (UID: \"08749638-af48-4615-9551-3b34e32233b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w" Oct 30 13:12:52.197350 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.185964 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:12:52.197494 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.188467 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w"] Oct 30 13:12:52.200459 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.185852 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:12:52.298531 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.298494 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwb5x\" (UniqueName: \"kubernetes.io/projected/08749638-af48-4615-9551-3b34e32233b3-kube-api-access-pwb5x\") pod \"maas-api-revoke-sub-345fe60c-mhg2w\" (UID: \"08749638-af48-4615-9551-3b34e32233b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w" Oct 30 13:12:52.298713 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.298613 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08749638-af48-4615-9551-3b34e32233b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-345fe60c-mhg2w\" (UID: \"08749638-af48-4615-9551-3b34e32233b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w" Oct 30 13:12:52.298770 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.298722 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08749638-af48-4615-9551-3b34e32233b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-345fe60c-mhg2w\" (UID: \"08749638-af48-4615-9551-3b34e32233b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w" Oct 30 13:12:52.299398 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.299371 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08749638-af48-4615-9551-3b34e32233b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-345fe60c-mhg2w\" (UID: \"08749638-af48-4615-9551-3b34e32233b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w" Oct 30 13:12:52.301076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.301054 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08749638-af48-4615-9551-3b34e32233b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-345fe60c-mhg2w\" (UID: \"08749638-af48-4615-9551-3b34e32233b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w" Oct 30 13:12:52.323039 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.323006 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwb5x\" (UniqueName: \"kubernetes.io/projected/08749638-af48-4615-9551-3b34e32233b3-kube-api-access-pwb5x\") pod \"maas-api-revoke-sub-345fe60c-mhg2w\" (UID: \"08749638-af48-4615-9551-3b34e32233b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w" Oct 30 13:12:52.518677 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.518648 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w" Oct 30 13:12:52.876454 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:52.876421 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w"] Oct 30 13:12:52.877213 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:12:52.877185 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08749638_af48_4615_9551_3b34e32233b3.slice/crio-9ae5daba69a9167dc2543d8e3849bd09766c358efa3273290595c1b1894f90e7 WatchSource:0}: Error finding container 9ae5daba69a9167dc2543d8e3849bd09766c358efa3273290595c1b1894f90e7: Status 404 returned error can't find the container with id 9ae5daba69a9167dc2543d8e3849bd09766c358efa3273290595c1b1894f90e7 Oct 30 13:12:53.730530 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:53.730497 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w" event={"ID":"08749638-af48-4615-9551-3b34e32233b3","Type":"ContainerStarted","Data":"1713a29c32a56ed21700c7c63291197a245fe83fa939a88af16b35c093efac05"} Oct 30 13:12:53.730990 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:53.730547 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w" event={"ID":"08749638-af48-4615-9551-3b34e32233b3","Type":"ContainerStarted","Data":"9ae5daba69a9167dc2543d8e3849bd09766c358efa3273290595c1b1894f90e7"} Oct 30 13:12:53.772973 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:53.772857 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w" podStartSLOduration=1.772839572 podStartE2EDuration="1.772839572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:53.771565796 +0000 UTC m=+1267.594794076" watchObservedRunningTime="2026-10-30 13:12:53.772839572 +0000 UTC m=+1267.596067850" Oct 30 13:12:54.738725 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:54.738689 4503 generic.go:356] "Generic (PLEG): container finished" podID="08749638-af48-4615-9551-3b34e32233b3" containerID="1713a29c32a56ed21700c7c63291197a245fe83fa939a88af16b35c093efac05" exitCode=0 Oct 30 13:12:54.739243 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:54.738774 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w" event={"ID":"08749638-af48-4615-9551-3b34e32233b3","Type":"ContainerDied","Data":"1713a29c32a56ed21700c7c63291197a245fe83fa939a88af16b35c093efac05"} Oct 30 13:12:55.041448 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:55.041369 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9"] Oct 30 13:12:55.041661 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:55.041635 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" podUID="692ff35a-d45a-4a6e-80f6-92511e76e7fd" containerName="istio-proxy" containerID="cri-o://b85fde405feeba3a867487bb397205d004ed700fe909419232ceadb662358960" gracePeriod=30 Oct 30 13:12:55.056110 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:55.055123 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb"] Oct 30 13:12:55.056110 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:55.055668 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" podUID="1a8afe3e-7c5d-4ba1-93a2-898433af86d7" containerName="istio-proxy" containerID="cri-o://120b4cc973950e7b36dc5995c281476a8bdd7bfdffa40f10d7c8459974fcf6d0" gracePeriod=30 Oct 30 13:12:55.767124 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:55.767057 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" podUID="9402ca3b-a5e0-49b8-83a2-e84331209e84" containerName="payload-processing" containerID="cri-o://32a5b2ed648298ecd1152492b95b401d0c3000ab4898cd45813534e06acb98ec" gracePeriod=25 Oct 30 13:12:55.902904 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:55.902878 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w" Oct 30 13:12:55.906225 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:55.906202 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08749638-af48-4615-9551-3b34e32233b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08749638-af48-4615-9551-3b34e32233b3-maas-api-xks-ca\") pod \"08749638-af48-4615-9551-3b34e32233b3\" (UID: \"08749638-af48-4615-9551-3b34e32233b3\") " Oct 30 13:12:55.906295 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:55.906262 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08749638-af48-4615-9551-3b34e32233b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08749638-af48-4615-9551-3b34e32233b3-maas-api-openshift-ca\") pod \"08749638-af48-4615-9551-3b34e32233b3\" (UID: \"08749638-af48-4615-9551-3b34e32233b3\") " Oct 30 13:12:55.906332 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:55.906314 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08749638-af48-4615-9551-3b34e32233b3-kube-api-access-pwb5x\" (UniqueName: \"kubernetes.io/projected/08749638-af48-4615-9551-3b34e32233b3-kube-api-access-pwb5x\") pod \"08749638-af48-4615-9551-3b34e32233b3\" (UID: \"08749638-af48-4615-9551-3b34e32233b3\") " Oct 30 13:12:55.906735 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:55.906705 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08749638-af48-4615-9551-3b34e32233b3-maas-api-openshift-ca" pod "08749638-af48-4615-9551-3b34e32233b3" (UID: "08749638-af48-4615-9551-3b34e32233b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:55.906910 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:55.906892 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08749638-af48-4615-9551-3b34e32233b3-maas-api-openshift-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:55.908399 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:55.908376 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08749638-af48-4615-9551-3b34e32233b3-maas-api-xks-ca" pod "08749638-af48-4615-9551-3b34e32233b3" (UID: "08749638-af48-4615-9551-3b34e32233b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:12:55.908499 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:55.908480 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08749638-af48-4615-9551-3b34e32233b3-kube-api-access-pwb5x" pod "08749638-af48-4615-9551-3b34e32233b3" (UID: "08749638-af48-4615-9551-3b34e32233b3"). InnerVolumeSpecName "kube-api-access-pwb5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:56.007803 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:56.007716 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08749638-af48-4615-9551-3b34e32233b3-maas-api-xks-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:56.007803 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:56.007748 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwb5x\" (UniqueName: \"kubernetes.io/projected/08749638-af48-4615-9551-3b34e32233b3-kube-api-access-pwb5x\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:12:56.872381 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:56.872279 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w" event={"ID":"08749638-af48-4615-9551-3b34e32233b3","Type":"ContainerDied","Data":"9ae5daba69a9167dc2543d8e3849bd09766c358efa3273290595c1b1894f90e7"} Oct 30 13:12:56.873955 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:56.873914 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ae5daba69a9167dc2543d8e3849bd09766c358efa3273290595c1b1894f90e7" Oct 30 13:12:56.874121 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:56.872987 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w" Oct 30 13:12:59.058493 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:59.058442 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:12:59.058959 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:12:59.058513 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" podUID="1a8afe3e-7c5d-4ba1-93a2-898433af86d7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:12:59.701579 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:12:59.701542 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:13:00.223014 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.222981 4503 patch_prober.go:29] interesting pod/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 13:13:00.223422 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.223037 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" podUID="692ff35a-d45a-4a6e-80f6-92511e76e7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 13:13:00.271031 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.266924 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" containerID="cri-o://d22d0988438c833e247c32d3a009645393a14fbba89fbca9f85a3f5bcf4cdc19" gracePeriod=45 Oct 30 13:13:00.280683 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.280639 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" containerID="cri-o://7c5bf279ce61edc176b59830a165f458bdf4ef3f0cd6cd47d7acce13aa6a27e7" gracePeriod=45 Oct 30 13:13:00.416116 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.416088 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:13:00.418873 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.418844 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:13:00.509462 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.509429 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istiod-ca-cert\") pod \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " Oct 30 13:13:00.509646 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.509501 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-data\") pod \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " Oct 30 13:13:00.509646 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.509593 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istiod-ca-cert\") pod \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " Oct 30 13:13:00.509646 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.509631 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-workload-socket\") pod \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " Oct 30 13:13:00.509821 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.509679 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-envoy\") pod \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " Oct 30 13:13:00.509821 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.509710 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-token\" (UniqueName: \"kubernetes.io/projected/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-token\") pod \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " Oct 30 13:13:00.509821 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.509756 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-workload-certs\") pod \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " Oct 30 13:13:00.509821 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.509802 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-data" pod "692ff35a-d45a-4a6e-80f6-92511e76e7fd" (UID: "692ff35a-d45a-4a6e-80f6-92511e76e7fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.509821 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.509809 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-podinfo\") pod \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " Oct 30 13:13:00.510120 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.509872 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-credential-socket\") pod \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " Oct 30 13:13:00.510120 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.509893 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istiod-ca-cert" pod "692ff35a-d45a-4a6e-80f6-92511e76e7fd" (UID: "692ff35a-d45a-4a6e-80f6-92511e76e7fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:00.510120 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.509908 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/692ff35a-d45a-4a6e-80f6-92511e76e7fd-kube-api-access-bdwg8\" (UniqueName: \"kubernetes.io/projected/692ff35a-d45a-4a6e-80f6-92511e76e7fd-kube-api-access-bdwg8\") pod \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " Oct 30 13:13:00.510120 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.510022 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-token\" (UniqueName: \"kubernetes.io/projected/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-token\") pod \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " Oct 30 13:13:00.510120 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.510082 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-podinfo\") pod \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " Oct 30 13:13:00.510382 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.510122 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-workload-socket\") pod \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " Oct 30 13:13:00.510382 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.510121 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-workload-certs" pod "692ff35a-d45a-4a6e-80f6-92511e76e7fd" (UID: "692ff35a-d45a-4a6e-80f6-92511e76e7fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.510382 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.510171 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-workload-certs\") pod \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " Oct 30 13:13:00.510382 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.510212 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-envoy\") pod \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " Oct 30 13:13:00.510382 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.510259 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-data\") pod \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " Oct 30 13:13:00.510382 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.510321 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-kube-api-access-p9tm2\" (UniqueName: \"kubernetes.io/projected/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-kube-api-access-p9tm2\") pod \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\" (UID: \"1a8afe3e-7c5d-4ba1-93a2-898433af86d7\") " Oct 30 13:13:00.510382 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.510346 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-workload-socket" pod "692ff35a-d45a-4a6e-80f6-92511e76e7fd" (UID: "692ff35a-d45a-4a6e-80f6-92511e76e7fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.510745 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.510410 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-credential-socket\") pod \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\" (UID: \"692ff35a-d45a-4a6e-80f6-92511e76e7fd\") " Oct 30 13:13:00.510745 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.510462 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-workload-socket" pod "1a8afe3e-7c5d-4ba1-93a2-898433af86d7" (UID: "1a8afe3e-7c5d-4ba1-93a2-898433af86d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.510745 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.510647 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-credential-socket" pod "1a8afe3e-7c5d-4ba1-93a2-898433af86d7" (UID: "1a8afe3e-7c5d-4ba1-93a2-898433af86d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.510901 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.510776 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-data" pod "1a8afe3e-7c5d-4ba1-93a2-898433af86d7" (UID: "1a8afe3e-7c5d-4ba1-93a2-898433af86d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.510901 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.510818 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-workload-certs" pod "1a8afe3e-7c5d-4ba1-93a2-898433af86d7" (UID: "1a8afe3e-7c5d-4ba1-93a2-898433af86d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.511103 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.511075 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.511163 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.511113 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.511163 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.511131 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.511163 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.511143 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.511163 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.511155 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.511366 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.511167 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.511366 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.511180 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.511366 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.511192 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.511366 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.510817 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-credential-socket" pod "692ff35a-d45a-4a6e-80f6-92511e76e7fd" (UID: "692ff35a-d45a-4a6e-80f6-92511e76e7fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.511366 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.511266 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istiod-ca-cert" pod "1a8afe3e-7c5d-4ba1-93a2-898433af86d7" (UID: "1a8afe3e-7c5d-4ba1-93a2-898433af86d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:00.513429 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.513382 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-token" pod "692ff35a-d45a-4a6e-80f6-92511e76e7fd" (UID: "692ff35a-d45a-4a6e-80f6-92511e76e7fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:00.514433 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.514397 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-podinfo" pod "692ff35a-d45a-4a6e-80f6-92511e76e7fd" (UID: "692ff35a-d45a-4a6e-80f6-92511e76e7fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:00.514687 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.514648 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-envoy" pod "1a8afe3e-7c5d-4ba1-93a2-898433af86d7" (UID: "1a8afe3e-7c5d-4ba1-93a2-898433af86d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.515200 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.515180 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-token" pod "1a8afe3e-7c5d-4ba1-93a2-898433af86d7" (UID: "1a8afe3e-7c5d-4ba1-93a2-898433af86d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:00.515388 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.515371 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-podinfo" pod "1a8afe3e-7c5d-4ba1-93a2-898433af86d7" (UID: "1a8afe3e-7c5d-4ba1-93a2-898433af86d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:00.517294 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.517263 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692ff35a-d45a-4a6e-80f6-92511e76e7fd-kube-api-access-bdwg8" pod "692ff35a-d45a-4a6e-80f6-92511e76e7fd" (UID: "692ff35a-d45a-4a6e-80f6-92511e76e7fd"). InnerVolumeSpecName "kube-api-access-bdwg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:00.517473 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.517438 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-kube-api-access-p9tm2" pod "1a8afe3e-7c5d-4ba1-93a2-898433af86d7" (UID: "1a8afe3e-7c5d-4ba1-93a2-898433af86d7"). InnerVolumeSpecName "kube-api-access-p9tm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:00.518585 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.518338 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-envoy" pod "692ff35a-d45a-4a6e-80f6-92511e76e7fd" (UID: "692ff35a-d45a-4a6e-80f6-92511e76e7fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.548502 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.548464 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:13:00.592163 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.592131 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.612350 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-home\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-home\") pod \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.612410 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2001219c-7af6-4dab-a7cc-ec2a30186f2e-kube-api-access-dfdbr\" (UniqueName: \"kubernetes.io/projected/2001219c-7af6-4dab-a7cc-ec2a30186f2e-kube-api-access-dfdbr\") pod \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.612462 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-tmp-dir\") pod \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.612487 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-dshm\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-dshm\") pod \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.612522 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-model-cache\") pod \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.612575 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-model-cache\") pod \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.612616 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2001219c-7af6-4dab-a7cc-ec2a30186f2e-tls-certs\" (UniqueName: \"kubernetes.io/secret/2001219c-7af6-4dab-a7cc-ec2a30186f2e-tls-certs\") pod \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.612647 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-dshm\") pod \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.612682 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30750a5b-e095-46ec-bb64-fbfc4bfa7427-tls-certs\" (UniqueName: \"kubernetes.io/secret/30750a5b-e095-46ec-bb64-fbfc4bfa7427-tls-certs\") pod \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.612723 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-tmp-dir\") pod \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.612755 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30750a5b-e095-46ec-bb64-fbfc4bfa7427-kube-api-access-qjb7l\" (UniqueName: \"kubernetes.io/projected/30750a5b-e095-46ec-bb64-fbfc4bfa7427-kube-api-access-qjb7l\") pod \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\" (UID: \"30750a5b-e095-46ec-bb64-fbfc4bfa7427\") " Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.612857 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-home\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-home\") pod \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\" (UID: \"2001219c-7af6-4dab-a7cc-ec2a30186f2e\") " Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.613303 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.613322 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.613330 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.613338 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.613346 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdwg8\" (UniqueName: \"kubernetes.io/projected/692ff35a-d45a-4a6e-80f6-92511e76e7fd-kube-api-access-bdwg8\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.613354 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/692ff35a-d45a-4a6e-80f6-92511e76e7fd-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.613363 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.613371 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.613380 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9tm2\" (UniqueName: \"kubernetes.io/projected/1a8afe3e-7c5d-4ba1-93a2-898433af86d7-kube-api-access-p9tm2\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.613759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.613388 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ff35a-d45a-4a6e-80f6-92511e76e7fd-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.615073 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.613784 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-tmp-dir" pod "30750a5b-e095-46ec-bb64-fbfc4bfa7427" (UID: "30750a5b-e095-46ec-bb64-fbfc4bfa7427"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.615073 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.613885 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-model-cache" pod "30750a5b-e095-46ec-bb64-fbfc4bfa7427" (UID: "30750a5b-e095-46ec-bb64-fbfc4bfa7427"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.615073 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.613959 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-home" pod "30750a5b-e095-46ec-bb64-fbfc4bfa7427" (UID: "30750a5b-e095-46ec-bb64-fbfc4bfa7427"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.615073 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.614135 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-tmp-dir" pod "2001219c-7af6-4dab-a7cc-ec2a30186f2e" (UID: "2001219c-7af6-4dab-a7cc-ec2a30186f2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.615073 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.614366 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-model-cache" pod "2001219c-7af6-4dab-a7cc-ec2a30186f2e" (UID: "2001219c-7af6-4dab-a7cc-ec2a30186f2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.615073 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.615017 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-home" pod "2001219c-7af6-4dab-a7cc-ec2a30186f2e" (UID: "2001219c-7af6-4dab-a7cc-ec2a30186f2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.626988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.618103 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-dshm" pod "2001219c-7af6-4dab-a7cc-ec2a30186f2e" (UID: "2001219c-7af6-4dab-a7cc-ec2a30186f2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.626988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.618333 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2001219c-7af6-4dab-a7cc-ec2a30186f2e-kube-api-access-dfdbr" pod "2001219c-7af6-4dab-a7cc-ec2a30186f2e" (UID: "2001219c-7af6-4dab-a7cc-ec2a30186f2e"). InnerVolumeSpecName "kube-api-access-dfdbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:00.626988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.618965 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-dshm" pod "30750a5b-e095-46ec-bb64-fbfc4bfa7427" (UID: "30750a5b-e095-46ec-bb64-fbfc4bfa7427"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.626988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.619079 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30750a5b-e095-46ec-bb64-fbfc4bfa7427-kube-api-access-qjb7l" pod "30750a5b-e095-46ec-bb64-fbfc4bfa7427" (UID: "30750a5b-e095-46ec-bb64-fbfc4bfa7427"). InnerVolumeSpecName "kube-api-access-qjb7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:00.626988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.620775 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2001219c-7af6-4dab-a7cc-ec2a30186f2e-tls-certs" pod "2001219c-7af6-4dab-a7cc-ec2a30186f2e" (UID: "2001219c-7af6-4dab-a7cc-ec2a30186f2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:13:00.626988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.621175 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30750a5b-e095-46ec-bb64-fbfc4bfa7427-tls-certs" pod "30750a5b-e095-46ec-bb64-fbfc4bfa7427" (UID: "30750a5b-e095-46ec-bb64-fbfc4bfa7427"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:13:00.626988 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.621852 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" podUID="9402ca3b-a5e0-49b8-83a2-e84331209e84" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:13:00.715230 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.715192 4503 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-home\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.715230 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.715232 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfdbr\" (UniqueName: \"kubernetes.io/projected/2001219c-7af6-4dab-a7cc-ec2a30186f2e-kube-api-access-dfdbr\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.715430 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.715248 4503 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-tmp-dir\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.715430 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.715265 4503 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-dshm\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.715430 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.715277 4503 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-model-cache\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.715430 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.715290 4503 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-model-cache\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.715430 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.715302 4503 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2001219c-7af6-4dab-a7cc-ec2a30186f2e-tls-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.715430 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.715314 4503 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-dshm\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.715430 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.715326 4503 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30750a5b-e095-46ec-bb64-fbfc4bfa7427-tls-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.715430 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.715338 4503 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30750a5b-e095-46ec-bb64-fbfc4bfa7427-tmp-dir\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.715430 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.715351 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjb7l\" (UniqueName: \"kubernetes.io/projected/30750a5b-e095-46ec-bb64-fbfc4bfa7427-kube-api-access-qjb7l\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.715430 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.715363 4503 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2001219c-7af6-4dab-a7cc-ec2a30186f2e-home\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.912291 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.912251 4503 generic.go:356] "Generic (PLEG): container finished" podID="692ff35a-d45a-4a6e-80f6-92511e76e7fd" containerID="b85fde405feeba3a867487bb397205d004ed700fe909419232ceadb662358960" exitCode=0 Oct 30 13:13:00.912486 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.912317 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" Oct 30 13:13:00.912486 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.912331 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" event={"ID":"692ff35a-d45a-4a6e-80f6-92511e76e7fd","Type":"ContainerDied","Data":"b85fde405feeba3a867487bb397205d004ed700fe909419232ceadb662358960"} Oct 30 13:13:00.912486 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.912422 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9" event={"ID":"692ff35a-d45a-4a6e-80f6-92511e76e7fd","Type":"ContainerDied","Data":"22ed4ee65cc0f5cf937148fa932c4c1d886eb3e73bd99b50396a4efabcdec9f8"} Oct 30 13:13:00.912486 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.912454 4503 scope.go:122] "RemoveContainer" containerID="b85fde405feeba3a867487bb397205d004ed700fe909419232ceadb662358960" Oct 30 13:13:00.913780 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.913749 4503 generic.go:356] "Generic (PLEG): container finished" podID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerID="7c5bf279ce61edc176b59830a165f458bdf4ef3f0cd6cd47d7acce13aa6a27e7" exitCode=0 Oct 30 13:13:00.913911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.913849 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" event={"ID":"30750a5b-e095-46ec-bb64-fbfc4bfa7427","Type":"ContainerDied","Data":"7c5bf279ce61edc176b59830a165f458bdf4ef3f0cd6cd47d7acce13aa6a27e7"} Oct 30 13:13:00.913911 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.913880 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" event={"ID":"30750a5b-e095-46ec-bb64-fbfc4bfa7427","Type":"ContainerDied","Data":"ac3b41e6630e0d63b34cfbaf1325c68da10a0148d69132ab3087cf87180c31d1"} Oct 30 13:13:00.914181 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.914166 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl" Oct 30 13:13:00.921718 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.921680 4503 generic.go:356] "Generic (PLEG): container finished" podID="1a8afe3e-7c5d-4ba1-93a2-898433af86d7" containerID="120b4cc973950e7b36dc5995c281476a8bdd7bfdffa40f10d7c8459974fcf6d0" exitCode=0 Oct 30 13:13:00.921870 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.921803 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" event={"ID":"1a8afe3e-7c5d-4ba1-93a2-898433af86d7","Type":"ContainerDied","Data":"120b4cc973950e7b36dc5995c281476a8bdd7bfdffa40f10d7c8459974fcf6d0"} Oct 30 13:13:00.921870 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.921851 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" event={"ID":"1a8afe3e-7c5d-4ba1-93a2-898433af86d7","Type":"ContainerDied","Data":"e88b0c3ebaa9302ca789aea7a2de6d05578b0640b37c4ee7870cff999805927f"} Oct 30 13:13:00.922022 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.921966 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb" Oct 30 13:13:00.925924 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.925894 4503 generic.go:356] "Generic (PLEG): container finished" podID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerID="d22d0988438c833e247c32d3a009645393a14fbba89fbca9f85a3f5bcf4cdc19" exitCode=0 Oct 30 13:13:00.926083 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.925950 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" Oct 30 13:13:00.926083 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.925992 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" event={"ID":"2001219c-7af6-4dab-a7cc-ec2a30186f2e","Type":"ContainerDied","Data":"d22d0988438c833e247c32d3a009645393a14fbba89fbca9f85a3f5bcf4cdc19"} Oct 30 13:13:00.926083 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.926027 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s" event={"ID":"2001219c-7af6-4dab-a7cc-ec2a30186f2e","Type":"ContainerDied","Data":"9b3e9513630b5f2e7baff45f74d4975d8e6ae18d83f6c9a385771255e239dad6"} Oct 30 13:13:00.927232 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.927216 4503 scope.go:122] "RemoveContainer" containerID="b85fde405feeba3a867487bb397205d004ed700fe909419232ceadb662358960" Oct 30 13:13:00.928209 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:13:00.927596 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b85fde405feeba3a867487bb397205d004ed700fe909419232ceadb662358960\": container with ID starting with b85fde405feeba3a867487bb397205d004ed700fe909419232ceadb662358960 not found: ID does not exist" containerID="b85fde405feeba3a867487bb397205d004ed700fe909419232ceadb662358960" Oct 30 13:13:00.928209 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.927632 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85fde405feeba3a867487bb397205d004ed700fe909419232ceadb662358960"} err="failed to get container status \"b85fde405feeba3a867487bb397205d004ed700fe909419232ceadb662358960\": rpc error: code = NotFound desc = could not find container \"b85fde405feeba3a867487bb397205d004ed700fe909419232ceadb662358960\": container with ID starting with b85fde405feeba3a867487bb397205d004ed700fe909419232ceadb662358960 not found: ID does not exist" Oct 30 13:13:00.928209 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.927655 4503 scope.go:122] "RemoveContainer" containerID="7c5bf279ce61edc176b59830a165f458bdf4ef3f0cd6cd47d7acce13aa6a27e7" Oct 30 13:13:00.939238 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.939218 4503 scope.go:122] "RemoveContainer" containerID="7c5bf279ce61edc176b59830a165f458bdf4ef3f0cd6cd47d7acce13aa6a27e7" Oct 30 13:13:00.939508 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:13:00.939486 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5bf279ce61edc176b59830a165f458bdf4ef3f0cd6cd47d7acce13aa6a27e7\": container with ID starting with 7c5bf279ce61edc176b59830a165f458bdf4ef3f0cd6cd47d7acce13aa6a27e7 not found: ID does not exist" containerID="7c5bf279ce61edc176b59830a165f458bdf4ef3f0cd6cd47d7acce13aa6a27e7" Oct 30 13:13:00.939599 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.939515 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5bf279ce61edc176b59830a165f458bdf4ef3f0cd6cd47d7acce13aa6a27e7"} err="failed to get container status \"7c5bf279ce61edc176b59830a165f458bdf4ef3f0cd6cd47d7acce13aa6a27e7\": rpc error: code = NotFound desc = could not find container \"7c5bf279ce61edc176b59830a165f458bdf4ef3f0cd6cd47d7acce13aa6a27e7\": container with ID starting with 7c5bf279ce61edc176b59830a165f458bdf4ef3f0cd6cd47d7acce13aa6a27e7 not found: ID does not exist" Oct 30 13:13:00.939599 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.939543 4503 scope.go:122] "RemoveContainer" containerID="120b4cc973950e7b36dc5995c281476a8bdd7bfdffa40f10d7c8459974fcf6d0" Oct 30 13:13:00.952077 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.952053 4503 scope.go:122] "RemoveContainer" containerID="120b4cc973950e7b36dc5995c281476a8bdd7bfdffa40f10d7c8459974fcf6d0" Oct 30 13:13:00.952420 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:13:00.952398 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"120b4cc973950e7b36dc5995c281476a8bdd7bfdffa40f10d7c8459974fcf6d0\": container with ID starting with 120b4cc973950e7b36dc5995c281476a8bdd7bfdffa40f10d7c8459974fcf6d0 not found: ID does not exist" containerID="120b4cc973950e7b36dc5995c281476a8bdd7bfdffa40f10d7c8459974fcf6d0" Oct 30 13:13:00.952521 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.952427 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"120b4cc973950e7b36dc5995c281476a8bdd7bfdffa40f10d7c8459974fcf6d0"} err="failed to get container status \"120b4cc973950e7b36dc5995c281476a8bdd7bfdffa40f10d7c8459974fcf6d0\": rpc error: code = NotFound desc = could not find container \"120b4cc973950e7b36dc5995c281476a8bdd7bfdffa40f10d7c8459974fcf6d0\": container with ID starting with 120b4cc973950e7b36dc5995c281476a8bdd7bfdffa40f10d7c8459974fcf6d0 not found: ID does not exist" Oct 30 13:13:00.952521 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.952445 4503 scope.go:122] "RemoveContainer" containerID="d22d0988438c833e247c32d3a009645393a14fbba89fbca9f85a3f5bcf4cdc19" Oct 30 13:13:00.961779 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.961759 4503 scope.go:122] "RemoveContainer" containerID="d22d0988438c833e247c32d3a009645393a14fbba89fbca9f85a3f5bcf4cdc19" Oct 30 13:13:00.962100 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:13:00.962078 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d22d0988438c833e247c32d3a009645393a14fbba89fbca9f85a3f5bcf4cdc19\": container with ID starting with d22d0988438c833e247c32d3a009645393a14fbba89fbca9f85a3f5bcf4cdc19 not found: ID does not exist" containerID="d22d0988438c833e247c32d3a009645393a14fbba89fbca9f85a3f5bcf4cdc19" Oct 30 13:13:00.962167 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:00.962110 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22d0988438c833e247c32d3a009645393a14fbba89fbca9f85a3f5bcf4cdc19"} err="failed to get container status \"d22d0988438c833e247c32d3a009645393a14fbba89fbca9f85a3f5bcf4cdc19\": rpc error: code = NotFound desc = could not find container \"d22d0988438c833e247c32d3a009645393a14fbba89fbca9f85a3f5bcf4cdc19\": container with ID starting with d22d0988438c833e247c32d3a009645393a14fbba89fbca9f85a3f5bcf4cdc19 not found: ID does not exist" Oct 30 13:13:01.024555 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:01.024519 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb"] Oct 30 13:13:01.037238 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:01.037204 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-qcsvb"] Oct 30 13:13:01.071158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:01.071117 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9"] Oct 30 13:13:01.078192 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:01.078026 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ac0c58-openshift-default-77ff8f86f5-656z9"] Oct 30 13:13:01.113865 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:01.113826 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl"] Oct 30 13:13:01.116839 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:01.116814 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-bb71e2/distinct-2-w2-bb71e2-kserve-5c8b4c6dc5-kcprl"] Oct 30 13:13:01.153585 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:01.153553 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s"] Oct 30 13:13:01.157048 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:01.157013 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-bb71e2/distinct-w2-bb71e2-kserve-7f4b84967b-24r9s"] Oct 30 13:13:02.387450 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:02.387383 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a8afe3e-7c5d-4ba1-93a2-898433af86d7" path="/var/lib/kubelet/pods/1a8afe3e-7c5d-4ba1-93a2-898433af86d7/volumes" Oct 30 13:13:02.393206 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:02.393173 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" path="/var/lib/kubelet/pods/2001219c-7af6-4dab-a7cc-ec2a30186f2e/volumes" Oct 30 13:13:02.398349 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:02.398316 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" path="/var/lib/kubelet/pods/30750a5b-e095-46ec-bb64-fbfc4bfa7427/volumes" Oct 30 13:13:02.399259 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:02.399234 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="692ff35a-d45a-4a6e-80f6-92511e76e7fd" path="/var/lib/kubelet/pods/692ff35a-d45a-4a6e-80f6-92511e76e7fd/volumes" Oct 30 13:13:02.652225 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:02.652192 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5"] Oct 30 13:13:02.652446 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:02.652421 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" podUID="f263d119-9ab0-440f-ac86-fa5a0d2a2819" containerName="istio-proxy" containerID="cri-o://ef7a4c1168ffbbdd707284db9cfa28b3d20fd3743f7d9e6bd856341ff435e7e4" gracePeriod=30 Oct 30 13:13:02.663876 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:02.663845 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5"] Oct 30 13:13:02.664135 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:02.664109 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" podUID="b366d790-421f-4e06-a5a9-5473df838ef5" containerName="istio-proxy" containerID="cri-o://86c7c0c2303b98548b7e6ceb7a46df90a2d0991bf8afdef66627778cba17574b" gracePeriod=30 Oct 30 13:13:02.672196 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:02.672166 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv"] Oct 30 13:13:02.672423 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:02.672400 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" podUID="51742836-262a-4345-ad56-bb8725b209ba" containerName="istio-proxy" containerID="cri-o://5352887cfd1c57622cf52ce5045f7a766c1a33d1662a56dd3533717120514ae3" gracePeriod=30 Oct 30 13:13:02.675926 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:02.675896 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw"] Oct 30 13:13:02.676167 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:02.676145 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" podUID="05e6b947-1515-4141-a00e-894c60bc30f1" containerName="istio-proxy" containerID="cri-o://9ce299a716a2721ee7e23efb1a658e127b5f1832db673418bc344bb1eddcbf4b" gracePeriod=30 Oct 30 13:13:06.846143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:06.846100 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc"] Oct 30 13:13:06.858368 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:06.858322 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bb71e2-jgstc"] Oct 30 13:13:07.981642 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:07.981602 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:13:07.985735 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:07.985576 4503 generic.go:356] "Generic (PLEG): container finished" podID="51742836-262a-4345-ad56-bb8725b209ba" containerID="5352887cfd1c57622cf52ce5045f7a766c1a33d1662a56dd3533717120514ae3" exitCode=0 Oct 30 13:13:07.985735 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:07.985664 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" event={"ID":"51742836-262a-4345-ad56-bb8725b209ba","Type":"ContainerDied","Data":"5352887cfd1c57622cf52ce5045f7a766c1a33d1662a56dd3533717120514ae3"} Oct 30 13:13:07.987903 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:07.987879 4503 generic.go:356] "Generic (PLEG): container finished" podID="b366d790-421f-4e06-a5a9-5473df838ef5" containerID="86c7c0c2303b98548b7e6ceb7a46df90a2d0991bf8afdef66627778cba17574b" exitCode=0 Oct 30 13:13:07.988062 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:07.988006 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" event={"ID":"b366d790-421f-4e06-a5a9-5473df838ef5","Type":"ContainerDied","Data":"86c7c0c2303b98548b7e6ceb7a46df90a2d0991bf8afdef66627778cba17574b"} Oct 30 13:13:07.988062 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:07.988051 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" event={"ID":"b366d790-421f-4e06-a5a9-5473df838ef5","Type":"ContainerDied","Data":"a03ce5565a872b7ad4182d2d1ceac40a427275e91a871f84896926f55419c68d"} Oct 30 13:13:07.988173 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:07.988080 4503 scope.go:122] "RemoveContainer" containerID="86c7c0c2303b98548b7e6ceb7a46df90a2d0991bf8afdef66627778cba17574b" Oct 30 13:13:07.988241 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:07.988221 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5" Oct 30 13:13:07.992439 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:07.992416 4503 generic.go:356] "Generic (PLEG): container finished" podID="05e6b947-1515-4141-a00e-894c60bc30f1" containerID="9ce299a716a2721ee7e23efb1a658e127b5f1832db673418bc344bb1eddcbf4b" exitCode=0 Oct 30 13:13:07.992654 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:07.992561 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" event={"ID":"05e6b947-1515-4141-a00e-894c60bc30f1","Type":"ContainerDied","Data":"9ce299a716a2721ee7e23efb1a658e127b5f1832db673418bc344bb1eddcbf4b"} Oct 30 13:13:07.995451 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:07.995141 4503 generic.go:356] "Generic (PLEG): container finished" podID="f263d119-9ab0-440f-ac86-fa5a0d2a2819" containerID="ef7a4c1168ffbbdd707284db9cfa28b3d20fd3743f7d9e6bd856341ff435e7e4" exitCode=0 Oct 30 13:13:07.995451 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:07.995231 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" event={"ID":"f263d119-9ab0-440f-ac86-fa5a0d2a2819","Type":"ContainerDied","Data":"ef7a4c1168ffbbdd707284db9cfa28b3d20fd3743f7d9e6bd856341ff435e7e4"} Oct 30 13:13:08.007444 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.007166 4503 scope.go:122] "RemoveContainer" containerID="86c7c0c2303b98548b7e6ceb7a46df90a2d0991bf8afdef66627778cba17574b" Oct 30 13:13:08.007712 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:13:08.007670 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c7c0c2303b98548b7e6ceb7a46df90a2d0991bf8afdef66627778cba17574b\": container with ID starting with 86c7c0c2303b98548b7e6ceb7a46df90a2d0991bf8afdef66627778cba17574b not found: ID does not exist" containerID="86c7c0c2303b98548b7e6ceb7a46df90a2d0991bf8afdef66627778cba17574b" Oct 30 13:13:08.007808 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.007728 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c7c0c2303b98548b7e6ceb7a46df90a2d0991bf8afdef66627778cba17574b"} err="failed to get container status \"86c7c0c2303b98548b7e6ceb7a46df90a2d0991bf8afdef66627778cba17574b\": rpc error: code = NotFound desc = could not find container \"86c7c0c2303b98548b7e6ceb7a46df90a2d0991bf8afdef66627778cba17574b\": container with ID starting with 86c7c0c2303b98548b7e6ceb7a46df90a2d0991bf8afdef66627778cba17574b not found: ID does not exist" Oct 30 13:13:08.007808 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.007754 4503 scope.go:122] "RemoveContainer" containerID="2912df6ca2092269d03f0287949e2156a18812e350585f9fbd2f9014230158c6" Oct 30 13:13:08.011832 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.011804 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:13:08.015841 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.015816 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:13:08.055907 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.055878 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-workload-certs\") pod \"b366d790-421f-4e06-a5a9-5473df838ef5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " Oct 30 13:13:08.057790 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.055973 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b366d790-421f-4e06-a5a9-5473df838ef5-istio-token\" (UniqueName: \"kubernetes.io/projected/b366d790-421f-4e06-a5a9-5473df838ef5-istio-token\") pod \"b366d790-421f-4e06-a5a9-5473df838ef5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " Oct 30 13:13:08.057790 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.056026 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-istio-data\") pod \"b366d790-421f-4e06-a5a9-5473df838ef5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " Oct 30 13:13:08.057790 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.056125 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-istio-envoy\") pod \"b366d790-421f-4e06-a5a9-5473df838ef5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " Oct 30 13:13:08.057790 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.056220 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b366d790-421f-4e06-a5a9-5473df838ef5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b366d790-421f-4e06-a5a9-5473df838ef5-istiod-ca-cert\") pod \"b366d790-421f-4e06-a5a9-5473df838ef5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " Oct 30 13:13:08.057790 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.056252 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-workload-socket\") pod \"b366d790-421f-4e06-a5a9-5473df838ef5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " Oct 30 13:13:08.057790 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.056307 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b366d790-421f-4e06-a5a9-5473df838ef5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b366d790-421f-4e06-a5a9-5473df838ef5-istio-podinfo\") pod \"b366d790-421f-4e06-a5a9-5473df838ef5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " Oct 30 13:13:08.057790 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.056381 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-credential-socket\") pod \"b366d790-421f-4e06-a5a9-5473df838ef5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " Oct 30 13:13:08.057790 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.056418 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b366d790-421f-4e06-a5a9-5473df838ef5-kube-api-access-56vch\" (UniqueName: \"kubernetes.io/projected/b366d790-421f-4e06-a5a9-5473df838ef5-kube-api-access-56vch\") pod \"b366d790-421f-4e06-a5a9-5473df838ef5\" (UID: \"b366d790-421f-4e06-a5a9-5473df838ef5\") " Oct 30 13:13:08.057790 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.056780 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-workload-certs" pod "b366d790-421f-4e06-a5a9-5473df838ef5" (UID: "b366d790-421f-4e06-a5a9-5473df838ef5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.057790 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.057062 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.057790 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.057297 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-istio-data" pod "b366d790-421f-4e06-a5a9-5473df838ef5" (UID: "b366d790-421f-4e06-a5a9-5473df838ef5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.057790 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.057394 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b366d790-421f-4e06-a5a9-5473df838ef5-istiod-ca-cert" pod "b366d790-421f-4e06-a5a9-5473df838ef5" (UID: "b366d790-421f-4e06-a5a9-5473df838ef5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:08.058807 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.058774 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-workload-socket" pod "b366d790-421f-4e06-a5a9-5473df838ef5" (UID: "b366d790-421f-4e06-a5a9-5473df838ef5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.058912 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.058840 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-credential-socket" pod "b366d790-421f-4e06-a5a9-5473df838ef5" (UID: "b366d790-421f-4e06-a5a9-5473df838ef5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.059008 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.058990 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:13:08.074478 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.062469 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b366d790-421f-4e06-a5a9-5473df838ef5-istio-podinfo" pod "b366d790-421f-4e06-a5a9-5473df838ef5" (UID: "b366d790-421f-4e06-a5a9-5473df838ef5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:08.074478 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.062684 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b366d790-421f-4e06-a5a9-5473df838ef5-kube-api-access-56vch" pod "b366d790-421f-4e06-a5a9-5473df838ef5" (UID: "b366d790-421f-4e06-a5a9-5473df838ef5"). InnerVolumeSpecName "kube-api-access-56vch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:08.074478 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.062835 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-istio-envoy" pod "b366d790-421f-4e06-a5a9-5473df838ef5" (UID: "b366d790-421f-4e06-a5a9-5473df838ef5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.075154 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.075114 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b366d790-421f-4e06-a5a9-5473df838ef5-istio-token" pod "b366d790-421f-4e06-a5a9-5473df838ef5" (UID: "b366d790-421f-4e06-a5a9-5473df838ef5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:08.158284 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158246 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-istio-envoy\") pod \"51742836-262a-4345-ad56-bb8725b209ba\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " Oct 30 13:13:08.158467 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158299 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05e6b947-1515-4141-a00e-894c60bc30f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e6b947-1515-4141-a00e-894c60bc30f1-istio-podinfo\") pod \"05e6b947-1515-4141-a00e-894c60bc30f1\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " Oct 30 13:13:08.158467 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158338 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-credential-socket\") pod \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " Oct 30 13:13:08.158467 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158364 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51742836-262a-4345-ad56-bb8725b209ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51742836-262a-4345-ad56-bb8725b209ba-istio-podinfo\") pod \"51742836-262a-4345-ad56-bb8725b209ba\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " Oct 30 13:13:08.158467 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158398 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51742836-262a-4345-ad56-bb8725b209ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51742836-262a-4345-ad56-bb8725b209ba-istiod-ca-cert\") pod \"51742836-262a-4345-ad56-bb8725b209ba\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " Oct 30 13:13:08.158467 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158428 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istiod-ca-cert\") pod \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " Oct 30 13:13:08.158805 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158469 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-workload-socket\") pod \"51742836-262a-4345-ad56-bb8725b209ba\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " Oct 30 13:13:08.158805 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158510 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-envoy\") pod \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " Oct 30 13:13:08.158805 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158543 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51742836-262a-4345-ad56-bb8725b209ba-istio-token\" (UniqueName: \"kubernetes.io/projected/51742836-262a-4345-ad56-bb8725b209ba-istio-token\") pod \"51742836-262a-4345-ad56-bb8725b209ba\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " Oct 30 13:13:08.158805 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158572 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-workload-certs\") pod \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " Oct 30 13:13:08.158805 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158599 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e6b947-1515-4141-a00e-894c60bc30f1-kube-api-access-2ck6l\" (UniqueName: \"kubernetes.io/projected/05e6b947-1515-4141-a00e-894c60bc30f1-kube-api-access-2ck6l\") pod \"05e6b947-1515-4141-a00e-894c60bc30f1\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " Oct 30 13:13:08.158805 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158658 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-podinfo\") pod \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " Oct 30 13:13:08.158805 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158689 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51742836-262a-4345-ad56-bb8725b209ba-kube-api-access-6jfvz\" (UniqueName: \"kubernetes.io/projected/51742836-262a-4345-ad56-bb8725b209ba-kube-api-access-6jfvz\") pod \"51742836-262a-4345-ad56-bb8725b209ba\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " Oct 30 13:13:08.158805 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158720 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-istio-data\") pod \"51742836-262a-4345-ad56-bb8725b209ba\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " Oct 30 13:13:08.158805 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158761 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f263d119-9ab0-440f-ac86-fa5a0d2a2819-kube-api-access-rsjgv\" (UniqueName: \"kubernetes.io/projected/f263d119-9ab0-440f-ac86-fa5a0d2a2819-kube-api-access-rsjgv\") pod \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " Oct 30 13:13:08.159276 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158809 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-credential-socket\") pod \"51742836-262a-4345-ad56-bb8725b209ba\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " Oct 30 13:13:08.159276 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158858 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-istio-envoy\") pod \"05e6b947-1515-4141-a00e-894c60bc30f1\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " Oct 30 13:13:08.159276 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158888 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-workload-certs\") pod \"05e6b947-1515-4141-a00e-894c60bc30f1\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " Oct 30 13:13:08.159276 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158917 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05e6b947-1515-4141-a00e-894c60bc30f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e6b947-1515-4141-a00e-894c60bc30f1-istiod-ca-cert\") pod \"05e6b947-1515-4141-a00e-894c60bc30f1\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " Oct 30 13:13:08.159276 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.158989 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-workload-certs\") pod \"51742836-262a-4345-ad56-bb8725b209ba\" (UID: \"51742836-262a-4345-ad56-bb8725b209ba\") " Oct 30 13:13:08.159276 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.159022 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-data\") pod \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " Oct 30 13:13:08.159276 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.159055 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-workload-socket\") pod \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " Oct 30 13:13:08.159276 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.159130 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-credential-socket\") pod \"05e6b947-1515-4141-a00e-894c60bc30f1\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " Oct 30 13:13:08.159276 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.159225 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-token\" (UniqueName: \"kubernetes.io/projected/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-token\") pod \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\" (UID: \"f263d119-9ab0-440f-ac86-fa5a0d2a2819\") " Oct 30 13:13:08.159846 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.159814 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-workload-socket" pod "51742836-262a-4345-ad56-bb8725b209ba" (UID: "51742836-262a-4345-ad56-bb8725b209ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.160202 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.160153 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-credential-socket" pod "f263d119-9ab0-440f-ac86-fa5a0d2a2819" (UID: "f263d119-9ab0-440f-ac86-fa5a0d2a2819"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.160954 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.160909 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-workload-socket\") pod \"05e6b947-1515-4141-a00e-894c60bc30f1\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " Oct 30 13:13:08.161047 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.160990 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-istio-data\") pod \"05e6b947-1515-4141-a00e-894c60bc30f1\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " Oct 30 13:13:08.161122 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.161097 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e6b947-1515-4141-a00e-894c60bc30f1-istio-token\" (UniqueName: \"kubernetes.io/projected/05e6b947-1515-4141-a00e-894c60bc30f1-istio-token\") pod \"05e6b947-1515-4141-a00e-894c60bc30f1\" (UID: \"05e6b947-1515-4141-a00e-894c60bc30f1\") " Oct 30 13:13:08.161985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.161660 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b366d790-421f-4e06-a5a9-5473df838ef5-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.161985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.161684 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.161985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.161699 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.161985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.161713 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b366d790-421f-4e06-a5a9-5473df838ef5-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.161985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.161726 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.161985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.161741 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b366d790-421f-4e06-a5a9-5473df838ef5-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.161985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.161757 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b366d790-421f-4e06-a5a9-5473df838ef5-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.161985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.161771 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56vch\" (UniqueName: \"kubernetes.io/projected/b366d790-421f-4e06-a5a9-5473df838ef5-kube-api-access-56vch\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.161985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.161786 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.161985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.161800 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.164709 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.162830 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-workload-certs" pod "51742836-262a-4345-ad56-bb8725b209ba" (UID: "51742836-262a-4345-ad56-bb8725b209ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.164709 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.163046 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-data" pod "f263d119-9ab0-440f-ac86-fa5a0d2a2819" (UID: "f263d119-9ab0-440f-ac86-fa5a0d2a2819"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.164709 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.164441 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-workload-socket" pod "f263d119-9ab0-440f-ac86-fa5a0d2a2819" (UID: "f263d119-9ab0-440f-ac86-fa5a0d2a2819"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.166631 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.165409 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-istio-data" pod "51742836-262a-4345-ad56-bb8725b209ba" (UID: "51742836-262a-4345-ad56-bb8725b209ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.166631 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.165786 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e6b947-1515-4141-a00e-894c60bc30f1-istio-token" pod "05e6b947-1515-4141-a00e-894c60bc30f1" (UID: "05e6b947-1515-4141-a00e-894c60bc30f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:08.166631 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.165801 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51742836-262a-4345-ad56-bb8725b209ba-istiod-ca-cert" pod "51742836-262a-4345-ad56-bb8725b209ba" (UID: "51742836-262a-4345-ad56-bb8725b209ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:08.166631 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.166062 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-workload-certs" pod "f263d119-9ab0-440f-ac86-fa5a0d2a2819" (UID: "f263d119-9ab0-440f-ac86-fa5a0d2a2819"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.166631 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.166216 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istiod-ca-cert" pod "f263d119-9ab0-440f-ac86-fa5a0d2a2819" (UID: "f263d119-9ab0-440f-ac86-fa5a0d2a2819"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:08.166631 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.166407 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-istio-data" pod "05e6b947-1515-4141-a00e-894c60bc30f1" (UID: "05e6b947-1515-4141-a00e-894c60bc30f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.166631 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.166461 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-credential-socket" pod "51742836-262a-4345-ad56-bb8725b209ba" (UID: "51742836-262a-4345-ad56-bb8725b209ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.167197 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.166652 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51742836-262a-4345-ad56-bb8725b209ba-istio-podinfo" pod "51742836-262a-4345-ad56-bb8725b209ba" (UID: "51742836-262a-4345-ad56-bb8725b209ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:08.167197 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.166699 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-workload-certs" pod "05e6b947-1515-4141-a00e-894c60bc30f1" (UID: "05e6b947-1515-4141-a00e-894c60bc30f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.168253 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.168230 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-workload-socket" pod "05e6b947-1515-4141-a00e-894c60bc30f1" (UID: "05e6b947-1515-4141-a00e-894c60bc30f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.168640 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.168622 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-credential-socket" pod "05e6b947-1515-4141-a00e-894c60bc30f1" (UID: "05e6b947-1515-4141-a00e-894c60bc30f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.168926 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.168891 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e6b947-1515-4141-a00e-894c60bc30f1-istiod-ca-cert" pod "05e6b947-1515-4141-a00e-894c60bc30f1" (UID: "05e6b947-1515-4141-a00e-894c60bc30f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:08.170258 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.170234 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-istio-envoy" pod "05e6b947-1515-4141-a00e-894c60bc30f1" (UID: "05e6b947-1515-4141-a00e-894c60bc30f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.172282 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.172188 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51742836-262a-4345-ad56-bb8725b209ba-istio-token" pod "51742836-262a-4345-ad56-bb8725b209ba" (UID: "51742836-262a-4345-ad56-bb8725b209ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:08.172282 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.172261 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e6b947-1515-4141-a00e-894c60bc30f1-kube-api-access-2ck6l" pod "05e6b947-1515-4141-a00e-894c60bc30f1" (UID: "05e6b947-1515-4141-a00e-894c60bc30f1"). InnerVolumeSpecName "kube-api-access-2ck6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:08.172478 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.172461 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05e6b947-1515-4141-a00e-894c60bc30f1-istio-podinfo" pod "05e6b947-1515-4141-a00e-894c60bc30f1" (UID: "05e6b947-1515-4141-a00e-894c60bc30f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:08.173097 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.173073 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-istio-envoy" pod "51742836-262a-4345-ad56-bb8725b209ba" (UID: "51742836-262a-4345-ad56-bb8725b209ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.173192 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.173115 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-token" pod "f263d119-9ab0-440f-ac86-fa5a0d2a2819" (UID: "f263d119-9ab0-440f-ac86-fa5a0d2a2819"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:08.173506 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.173484 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f263d119-9ab0-440f-ac86-fa5a0d2a2819-kube-api-access-rsjgv" pod "f263d119-9ab0-440f-ac86-fa5a0d2a2819" (UID: "f263d119-9ab0-440f-ac86-fa5a0d2a2819"). InnerVolumeSpecName "kube-api-access-rsjgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:08.173716 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.173689 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-podinfo" pod "f263d119-9ab0-440f-ac86-fa5a0d2a2819" (UID: "f263d119-9ab0-440f-ac86-fa5a0d2a2819"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:08.174206 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.174186 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-envoy" pod "f263d119-9ab0-440f-ac86-fa5a0d2a2819" (UID: "f263d119-9ab0-440f-ac86-fa5a0d2a2819"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:08.174443 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.174423 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51742836-262a-4345-ad56-bb8725b209ba-kube-api-access-6jfvz" pod "51742836-262a-4345-ad56-bb8725b209ba" (UID: "51742836-262a-4345-ad56-bb8725b209ba"). InnerVolumeSpecName "kube-api-access-6jfvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:08.262733 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262651 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e6b947-1515-4141-a00e-894c60bc30f1-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.262733 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262680 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.262733 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262690 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e6b947-1515-4141-a00e-894c60bc30f1-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.262733 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262699 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51742836-262a-4345-ad56-bb8725b209ba-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.262733 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262707 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51742836-262a-4345-ad56-bb8725b209ba-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.262733 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262717 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.262733 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262724 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.262733 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262731 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51742836-262a-4345-ad56-bb8725b209ba-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.262733 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262739 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.263158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262747 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ck6l\" (UniqueName: \"kubernetes.io/projected/05e6b947-1515-4141-a00e-894c60bc30f1-kube-api-access-2ck6l\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.263158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262756 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.263158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262764 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jfvz\" (UniqueName: \"kubernetes.io/projected/51742836-262a-4345-ad56-bb8725b209ba-kube-api-access-6jfvz\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.263158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262772 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.263158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262780 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsjgv\" (UniqueName: \"kubernetes.io/projected/f263d119-9ab0-440f-ac86-fa5a0d2a2819-kube-api-access-rsjgv\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.263158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262788 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.263158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262797 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.263158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262805 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.263158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262813 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e6b947-1515-4141-a00e-894c60bc30f1-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.263158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262820 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51742836-262a-4345-ad56-bb8725b209ba-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.263158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262827 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.263158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262834 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f263d119-9ab0-440f-ac86-fa5a0d2a2819-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.263158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262842 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.263158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262849 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f263d119-9ab0-440f-ac86-fa5a0d2a2819-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.263158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262856 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.263158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.262864 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e6b947-1515-4141-a00e-894c60bc30f1-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:08.338899 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.338864 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5"] Oct 30 13:13:08.347273 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.347240 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-xm8f5"] Oct 30 13:13:08.362798 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.362761 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57516645-0006-4f61-b699-dbc55090aecb" path="/var/lib/kubelet/pods/57516645-0006-4f61-b699-dbc55090aecb/volumes" Oct 30 13:13:08.363271 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:08.363257 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b366d790-421f-4e06-a5a9-5473df838ef5" path="/var/lib/kubelet/pods/b366d790-421f-4e06-a5a9-5473df838ef5/volumes" Oct 30 13:13:09.004316 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:09.004233 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" event={"ID":"51742836-262a-4345-ad56-bb8725b209ba","Type":"ContainerDied","Data":"91ee2e8e4584535a776bd5cc6a418c238be7c48174e69310cff7be3be57c576d"} Oct 30 13:13:09.004316 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:09.004265 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv" Oct 30 13:13:09.004316 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:09.004298 4503 scope.go:122] "RemoveContainer" containerID="5352887cfd1c57622cf52ce5045f7a766c1a33d1662a56dd3533717120514ae3" Oct 30 13:13:09.007739 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:09.007720 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" Oct 30 13:13:09.007881 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:09.007747 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw" event={"ID":"05e6b947-1515-4141-a00e-894c60bc30f1","Type":"ContainerDied","Data":"b3ea7fcc17403feb4195f3617687b5e93768b3b0e6256f51edde0ca3a9564873"} Oct 30 13:13:09.009301 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:09.009280 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" event={"ID":"f263d119-9ab0-440f-ac86-fa5a0d2a2819","Type":"ContainerDied","Data":"db2ae9cbd9d9ce17de76366f5a73d23f1f4eefb4ca76c12dee3c9bf493c66b68"} Oct 30 13:13:09.009422 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:09.009328 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5" Oct 30 13:13:09.021112 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:09.021087 4503 scope.go:122] "RemoveContainer" containerID="9ce299a716a2721ee7e23efb1a658e127b5f1832db673418bc344bb1eddcbf4b" Oct 30 13:13:09.031422 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:09.031400 4503 scope.go:122] "RemoveContainer" containerID="ef7a4c1168ffbbdd707284db9cfa28b3d20fd3743f7d9e6bd856341ff435e7e4" Oct 30 13:13:09.093980 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:09.093925 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5"] Oct 30 13:13:09.100030 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:09.099997 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-vsbc5"] Oct 30 13:13:09.135405 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:09.135370 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv"] Oct 30 13:13:09.140927 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:09.140891 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-cw2jv"] Oct 30 13:13:09.177759 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:09.177719 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw"] Oct 30 13:13:09.184764 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:09.184731 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-54994c-openshift-default-c8547749c-jqkvw"] Oct 30 13:13:10.366824 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:10.366789 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05e6b947-1515-4141-a00e-894c60bc30f1" path="/var/lib/kubelet/pods/05e6b947-1515-4141-a00e-894c60bc30f1/volumes" Oct 30 13:13:10.367435 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:10.367414 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51742836-262a-4345-ad56-bb8725b209ba" path="/var/lib/kubelet/pods/51742836-262a-4345-ad56-bb8725b209ba/volumes" Oct 30 13:13:10.367993 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:10.367974 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f263d119-9ab0-440f-ac86-fa5a0d2a2819" path="/var/lib/kubelet/pods/f263d119-9ab0-440f-ac86-fa5a0d2a2819/volumes" Oct 30 13:13:10.619683 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:10.619590 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" podUID="9402ca3b-a5e0-49b8-83a2-e84331209e84" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:13:15.705689 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:15.705652 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm"] Oct 30 13:13:16.038039 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.038016 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" Oct 30 13:13:16.074179 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.074147 4503 generic.go:356] "Generic (PLEG): container finished" podID="9402ca3b-a5e0-49b8-83a2-e84331209e84" containerID="32a5b2ed648298ecd1152492b95b401d0c3000ab4898cd45813534e06acb98ec" exitCode=0 Oct 30 13:13:16.074365 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.074230 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" Oct 30 13:13:16.074365 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.074234 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" event={"ID":"9402ca3b-a5e0-49b8-83a2-e84331209e84","Type":"ContainerDied","Data":"32a5b2ed648298ecd1152492b95b401d0c3000ab4898cd45813534e06acb98ec"} Oct 30 13:13:16.074365 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.074327 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr" event={"ID":"9402ca3b-a5e0-49b8-83a2-e84331209e84","Type":"ContainerDied","Data":"5191541214c3fe1989d8ba4d2e614e17da8d8f3e5a411cffcaec61d9fdebce93"} Oct 30 13:13:16.074365 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.074355 4503 scope.go:122] "RemoveContainer" containerID="32a5b2ed648298ecd1152492b95b401d0c3000ab4898cd45813534e06acb98ec" Oct 30 13:13:16.087442 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.087417 4503 scope.go:122] "RemoveContainer" containerID="32a5b2ed648298ecd1152492b95b401d0c3000ab4898cd45813534e06acb98ec" Oct 30 13:13:16.087855 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:13:16.087823 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a5b2ed648298ecd1152492b95b401d0c3000ab4898cd45813534e06acb98ec\": container with ID starting with 32a5b2ed648298ecd1152492b95b401d0c3000ab4898cd45813534e06acb98ec not found: ID does not exist" containerID="32a5b2ed648298ecd1152492b95b401d0c3000ab4898cd45813534e06acb98ec" Oct 30 13:13:16.087968 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.087867 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a5b2ed648298ecd1152492b95b401d0c3000ab4898cd45813534e06acb98ec"} err="failed to get container status \"32a5b2ed648298ecd1152492b95b401d0c3000ab4898cd45813534e06acb98ec\": rpc error: code = NotFound desc = could not find container \"32a5b2ed648298ecd1152492b95b401d0c3000ab4898cd45813534e06acb98ec\": container with ID starting with 32a5b2ed648298ecd1152492b95b401d0c3000ab4898cd45813534e06acb98ec not found: ID does not exist" Oct 30 13:13:16.166682 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.166651 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9402ca3b-a5e0-49b8-83a2-e84331209e84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9402ca3b-a5e0-49b8-83a2-e84331209e84-plugins-config-volume\") pod \"9402ca3b-a5e0-49b8-83a2-e84331209e84\" (UID: \"9402ca3b-a5e0-49b8-83a2-e84331209e84\") " Oct 30 13:13:16.167046 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.167016 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9402ca3b-a5e0-49b8-83a2-e84331209e84-plugins-config-volume" pod "9402ca3b-a5e0-49b8-83a2-e84331209e84" (UID: "9402ca3b-a5e0-49b8-83a2-e84331209e84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:16.167235 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.167178 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9402ca3b-a5e0-49b8-83a2-e84331209e84-kube-api-access-vspbp\" (UniqueName: \"kubernetes.io/projected/9402ca3b-a5e0-49b8-83a2-e84331209e84-kube-api-access-vspbp\") pod \"9402ca3b-a5e0-49b8-83a2-e84331209e84\" (UID: \"9402ca3b-a5e0-49b8-83a2-e84331209e84\") " Oct 30 13:13:16.167762 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.167729 4503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9402ca3b-a5e0-49b8-83a2-e84331209e84-plugins-config-volume\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:16.170504 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.170475 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9402ca3b-a5e0-49b8-83a2-e84331209e84-kube-api-access-vspbp" pod "9402ca3b-a5e0-49b8-83a2-e84331209e84" (UID: "9402ca3b-a5e0-49b8-83a2-e84331209e84"). InnerVolumeSpecName "kube-api-access-vspbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:16.269362 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.269272 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vspbp\" (UniqueName: \"kubernetes.io/projected/9402ca3b-a5e0-49b8-83a2-e84331209e84-kube-api-access-vspbp\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:16.391976 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.381590 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29"] Oct 30 13:13:16.391976 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.381869 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" podUID="88155745-668a-4d1c-8114-2c9eb2a13dba" containerName="istio-proxy" containerID="cri-o://a4c7b83daa8ef389d5976f326928a58d0fcbb4c86cffaf6c5bf486d82e71a448" gracePeriod=30 Oct 30 13:13:16.401284 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.401253 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb"] Oct 30 13:13:16.401565 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.401507 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerName="istio-proxy" containerID="cri-o://950cdbe3dbb332f000742d6b98c34775166e3efde6b0e30622cbb12a2e0eb3eb" gracePeriod=30 Oct 30 13:13:16.404326 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.404299 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g"] Oct 30 13:13:16.404613 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.404583 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" podUID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" containerName="istio-proxy" containerID="cri-o://dbff5914e675f231f498e2360270d32d3712e2debf16db9d305e229a1ce35845" gracePeriod=30 Oct 30 13:13:16.411388 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.411365 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw"] Oct 30 13:13:16.411546 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.411519 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" podUID="b8e7baf7-ef26-41d4-9dec-f4987791862d" containerName="istio-proxy" containerID="cri-o://0ba68ef1b81e853d282700ae458baef67793bc3b8b558c1f749a9a56885662f0" gracePeriod=30 Oct 30 13:13:16.592979 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.592884 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr"] Oct 30 13:13:16.600339 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:16.600308 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-bb71e2-5987786884-l68tr"] Oct 30 13:13:18.361698 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:18.361654 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9402ca3b-a5e0-49b8-83a2-e84331209e84" path="/var/lib/kubelet/pods/9402ca3b-a5e0-49b8-83a2-e84331209e84/volumes" Oct 30 13:13:21.376699 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.376662 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr"] Oct 30 13:13:21.720689 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.720653 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7"] Oct 30 13:13:21.721642 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.721617 4503 cpu_manager.go:414] "containerMap: removing container" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" Oct 30 13:13:21.721792 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.721779 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" Oct 30 13:13:21.721882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.721872 4503 cpu_manager.go:414] "containerMap: removing container" podUID="692ff35a-d45a-4a6e-80f6-92511e76e7fd" containerName="istio-proxy" Oct 30 13:13:21.722055 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.721972 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="692ff35a-d45a-4a6e-80f6-92511e76e7fd" containerName="istio-proxy" Oct 30 13:13:21.722055 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722007 4503 cpu_manager.go:414] "containerMap: removing container" podUID="51742836-262a-4345-ad56-bb8725b209ba" containerName="istio-proxy" Oct 30 13:13:21.722055 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722017 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51742836-262a-4345-ad56-bb8725b209ba" containerName="istio-proxy" Oct 30 13:13:21.722055 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722043 4503 cpu_manager.go:414] "containerMap: removing container" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" Oct 30 13:13:21.722055 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722051 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" Oct 30 13:13:21.722351 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722078 4503 cpu_manager.go:414] "containerMap: removing container" podUID="f263d119-9ab0-440f-ac86-fa5a0d2a2819" containerName="istio-proxy" Oct 30 13:13:21.722351 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722085 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f263d119-9ab0-440f-ac86-fa5a0d2a2819" containerName="istio-proxy" Oct 30 13:13:21.722351 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722104 4503 cpu_manager.go:414] "containerMap: removing container" podUID="1a8afe3e-7c5d-4ba1-93a2-898433af86d7" containerName="istio-proxy" Oct 30 13:13:21.722351 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722111 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8afe3e-7c5d-4ba1-93a2-898433af86d7" containerName="istio-proxy" Oct 30 13:13:21.722351 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722122 4503 cpu_manager.go:414] "containerMap: removing container" podUID="9402ca3b-a5e0-49b8-83a2-e84331209e84" containerName="payload-processing" Oct 30 13:13:21.722351 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722128 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9402ca3b-a5e0-49b8-83a2-e84331209e84" containerName="payload-processing" Oct 30 13:13:21.722351 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722140 4503 cpu_manager.go:414] "containerMap: removing container" podUID="f263d119-9ab0-440f-ac86-fa5a0d2a2819" containerName="istio-proxy" Oct 30 13:13:21.722351 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722147 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f263d119-9ab0-440f-ac86-fa5a0d2a2819" containerName="istio-proxy" Oct 30 13:13:21.722351 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722161 4503 cpu_manager.go:414] "containerMap: removing container" podUID="05e6b947-1515-4141-a00e-894c60bc30f1" containerName="istio-proxy" Oct 30 13:13:21.722351 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722168 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e6b947-1515-4141-a00e-894c60bc30f1" containerName="istio-proxy" Oct 30 13:13:21.722351 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722221 4503 cpu_manager.go:414] "containerMap: removing container" podUID="08749638-af48-4615-9551-3b34e32233b3" containerName="revoke-keys" Oct 30 13:13:21.722351 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722228 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08749638-af48-4615-9551-3b34e32233b3" containerName="revoke-keys" Oct 30 13:13:21.722351 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722239 4503 cpu_manager.go:414] "containerMap: removing container" podUID="b366d790-421f-4e06-a5a9-5473df838ef5" containerName="istio-proxy" Oct 30 13:13:21.722351 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722245 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b366d790-421f-4e06-a5a9-5473df838ef5" containerName="istio-proxy" Oct 30 13:13:21.722796 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722366 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="692ff35a-d45a-4a6e-80f6-92511e76e7fd" containerName="istio-proxy" Oct 30 13:13:21.722796 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722380 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="51742836-262a-4345-ad56-bb8725b209ba" containerName="istio-proxy" Oct 30 13:13:21.722796 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722392 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="30750a5b-e095-46ec-bb64-fbfc4bfa7427" containerName="main" Oct 30 13:13:21.722796 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722404 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="f263d119-9ab0-440f-ac86-fa5a0d2a2819" containerName="istio-proxy" Oct 30 13:13:21.722796 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722414 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8afe3e-7c5d-4ba1-93a2-898433af86d7" containerName="istio-proxy" Oct 30 13:13:21.722796 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722422 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="9402ca3b-a5e0-49b8-83a2-e84331209e84" containerName="payload-processing" Oct 30 13:13:21.722796 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722430 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="f263d119-9ab0-440f-ac86-fa5a0d2a2819" containerName="istio-proxy" Oct 30 13:13:21.722796 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722438 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e6b947-1515-4141-a00e-894c60bc30f1" containerName="istio-proxy" Oct 30 13:13:21.722796 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722447 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="08749638-af48-4615-9551-3b34e32233b3" containerName="revoke-keys" Oct 30 13:13:21.722796 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722457 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="b366d790-421f-4e06-a5a9-5473df838ef5" containerName="istio-proxy" Oct 30 13:13:21.722796 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.722466 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="2001219c-7af6-4dab-a7cc-ec2a30186f2e" containerName="main" Oct 30 13:13:21.729235 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.729209 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7" Oct 30 13:13:21.747147 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.743271 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7"] Oct 30 13:13:21.760832 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.760792 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xbwx\" (UniqueName: \"kubernetes.io/projected/c8d6f01a-f566-41ec-8d49-5dbc377ea951-kube-api-access-9xbwx\") pod \"maas-api-revoke-sub-88b26747-r5rm7\" (UID: \"c8d6f01a-f566-41ec-8d49-5dbc377ea951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7" Oct 30 13:13:21.761010 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.760893 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d6f01a-f566-41ec-8d49-5dbc377ea951-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88b26747-r5rm7\" (UID: \"c8d6f01a-f566-41ec-8d49-5dbc377ea951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7" Oct 30 13:13:21.761010 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.760929 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d6f01a-f566-41ec-8d49-5dbc377ea951-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88b26747-r5rm7\" (UID: \"c8d6f01a-f566-41ec-8d49-5dbc377ea951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7" Oct 30 13:13:21.772004 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.770983 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:13:21.819964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.815985 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:13:21.819964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.816061 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:13:21.858175 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.858146 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:13:21.861667 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.861638 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:13:21.861870 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.861828 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-credential-socket\") pod \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " Oct 30 13:13:21.861985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.861891 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-token\" (UniqueName: \"kubernetes.io/projected/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-token\") pod \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " Oct 30 13:13:21.861985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.861933 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-envoy\") pod \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " Oct 30 13:13:21.862121 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.862101 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-data\") pod \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " Oct 30 13:13:21.863290 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.863272 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-workload-socket\") pod \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " Oct 30 13:13:21.863483 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.863438 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istiod-ca-cert\") pod \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " Oct 30 13:13:21.863623 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.863608 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-podinfo\") pod \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " Oct 30 13:13:21.863726 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.863713 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-kube-api-access-4rpln\" (UniqueName: \"kubernetes.io/projected/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-kube-api-access-4rpln\") pod \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " Oct 30 13:13:21.863827 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.863813 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-workload-certs\") pod \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\" (UID: \"2c8e76b7-c2ba-4dd2-8b49-2d4565580919\") " Oct 30 13:13:21.864141 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.864120 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xbwx\" (UniqueName: \"kubernetes.io/projected/c8d6f01a-f566-41ec-8d49-5dbc377ea951-kube-api-access-9xbwx\") pod \"maas-api-revoke-sub-88b26747-r5rm7\" (UID: \"c8d6f01a-f566-41ec-8d49-5dbc377ea951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7" Oct 30 13:13:21.864348 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.864332 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d6f01a-f566-41ec-8d49-5dbc377ea951-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88b26747-r5rm7\" (UID: \"c8d6f01a-f566-41ec-8d49-5dbc377ea951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7" Oct 30 13:13:21.864633 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.864614 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d6f01a-f566-41ec-8d49-5dbc377ea951-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88b26747-r5rm7\" (UID: \"c8d6f01a-f566-41ec-8d49-5dbc377ea951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7" Oct 30 13:13:21.865208 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.862515 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-credential-socket" pod "2c8e76b7-c2ba-4dd2-8b49-2d4565580919" (UID: "2c8e76b7-c2ba-4dd2-8b49-2d4565580919"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.865300 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.863206 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-data" pod "2c8e76b7-c2ba-4dd2-8b49-2d4565580919" (UID: "2c8e76b7-c2ba-4dd2-8b49-2d4565580919"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.865300 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.863759 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istiod-ca-cert" pod "2c8e76b7-c2ba-4dd2-8b49-2d4565580919" (UID: "2c8e76b7-c2ba-4dd2-8b49-2d4565580919"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:21.865300 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.864071 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-workload-socket" pod "2c8e76b7-c2ba-4dd2-8b49-2d4565580919" (UID: "2c8e76b7-c2ba-4dd2-8b49-2d4565580919"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.865300 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.864907 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-token" pod "2c8e76b7-c2ba-4dd2-8b49-2d4565580919" (UID: "2c8e76b7-c2ba-4dd2-8b49-2d4565580919"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:21.865300 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.865162 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-workload-certs" pod "2c8e76b7-c2ba-4dd2-8b49-2d4565580919" (UID: "2c8e76b7-c2ba-4dd2-8b49-2d4565580919"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.866344 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.866323 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d6f01a-f566-41ec-8d49-5dbc377ea951-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88b26747-r5rm7\" (UID: \"c8d6f01a-f566-41ec-8d49-5dbc377ea951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7" Oct 30 13:13:21.867216 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.867192 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-envoy" pod "2c8e76b7-c2ba-4dd2-8b49-2d4565580919" (UID: "2c8e76b7-c2ba-4dd2-8b49-2d4565580919"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.869136 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.868837 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-podinfo" pod "2c8e76b7-c2ba-4dd2-8b49-2d4565580919" (UID: "2c8e76b7-c2ba-4dd2-8b49-2d4565580919"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:21.869136 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.869087 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-kube-api-access-4rpln" pod "2c8e76b7-c2ba-4dd2-8b49-2d4565580919" (UID: "2c8e76b7-c2ba-4dd2-8b49-2d4565580919"). InnerVolumeSpecName "kube-api-access-4rpln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:21.869810 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.869597 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d6f01a-f566-41ec-8d49-5dbc377ea951-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88b26747-r5rm7\" (UID: \"c8d6f01a-f566-41ec-8d49-5dbc377ea951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7" Oct 30 13:13:21.874205 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.874184 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:13:21.892440 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.892412 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xbwx\" (UniqueName: \"kubernetes.io/projected/c8d6f01a-f566-41ec-8d49-5dbc377ea951-kube-api-access-9xbwx\") pod \"maas-api-revoke-sub-88b26747-r5rm7\" (UID: \"c8d6f01a-f566-41ec-8d49-5dbc377ea951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7" Oct 30 13:13:21.966346 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966255 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-workload-socket\") pod \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " Oct 30 13:13:21.966346 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966335 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-token\" (UniqueName: \"kubernetes.io/projected/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-token\") pod \"b8e7baf7-ef26-41d4-9dec-f4987791862d\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " Oct 30 13:13:21.966571 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966372 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-envoy\") pod \"b8e7baf7-ef26-41d4-9dec-f4987791862d\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " Oct 30 13:13:21.966571 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966411 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-credential-socket\") pod \"b8e7baf7-ef26-41d4-9dec-f4987791862d\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " Oct 30 13:13:21.966571 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966458 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8e7baf7-ef26-41d4-9dec-f4987791862d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8e7baf7-ef26-41d4-9dec-f4987791862d-istiod-ca-cert\") pod \"b8e7baf7-ef26-41d4-9dec-f4987791862d\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " Oct 30 13:13:21.966571 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966496 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-data\") pod \"88155745-668a-4d1c-8114-2c9eb2a13dba\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " Oct 30 13:13:21.966571 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966534 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-envoy\") pod \"88155745-668a-4d1c-8114-2c9eb2a13dba\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " Oct 30 13:13:21.966571 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966563 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88155745-668a-4d1c-8114-2c9eb2a13dba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88155745-668a-4d1c-8114-2c9eb2a13dba-istiod-ca-cert\") pod \"88155745-668a-4d1c-8114-2c9eb2a13dba\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " Oct 30 13:13:21.966830 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966576 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-workload-socket" pod "41e428c4-45aa-4d5d-b6ea-7939b76b2831" (UID: "41e428c4-45aa-4d5d-b6ea-7939b76b2831"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.966830 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966613 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-token\" (UniqueName: \"kubernetes.io/projected/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-token\") pod \"88155745-668a-4d1c-8114-2c9eb2a13dba\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " Oct 30 13:13:21.966830 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966641 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-workload-certs\") pod \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " Oct 30 13:13:21.966830 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966676 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-podinfo\") pod \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " Oct 30 13:13:21.966830 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966702 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-credential-socket\") pod \"88155745-668a-4d1c-8114-2c9eb2a13dba\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " Oct 30 13:13:21.966830 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966730 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-podinfo\") pod \"88155745-668a-4d1c-8114-2c9eb2a13dba\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " Oct 30 13:13:21.966830 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966759 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e428c4-45aa-4d5d-b6ea-7939b76b2831-kube-api-access-xcvnh\" (UniqueName: \"kubernetes.io/projected/41e428c4-45aa-4d5d-b6ea-7939b76b2831-kube-api-access-xcvnh\") pod \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " Oct 30 13:13:21.966830 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966791 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-workload-socket\") pod \"b8e7baf7-ef26-41d4-9dec-f4987791862d\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " Oct 30 13:13:21.966830 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966830 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88155745-668a-4d1c-8114-2c9eb2a13dba-kube-api-access-nlrnh\" (UniqueName: \"kubernetes.io/projected/88155745-668a-4d1c-8114-2c9eb2a13dba-kube-api-access-nlrnh\") pod \"88155745-668a-4d1c-8114-2c9eb2a13dba\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " Oct 30 13:13:21.967145 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966891 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-credential-socket\") pod \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " Oct 30 13:13:21.967145 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966926 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e7baf7-ef26-41d4-9dec-f4987791862d-istiod-ca-cert" pod "b8e7baf7-ef26-41d4-9dec-f4987791862d" (UID: "b8e7baf7-ef26-41d4-9dec-f4987791862d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:21.967145 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966931 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-envoy\") pod \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " Oct 30 13:13:21.967145 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.966994 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-data\") pod \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " Oct 30 13:13:21.967145 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.967028 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istiod-ca-cert\") pod \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " Oct 30 13:13:21.967145 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.967073 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-data\") pod \"b8e7baf7-ef26-41d4-9dec-f4987791862d\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " Oct 30 13:13:21.967145 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.967118 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-podinfo\") pod \"b8e7baf7-ef26-41d4-9dec-f4987791862d\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " Oct 30 13:13:21.967405 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.967146 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-workload-certs\") pod \"88155745-668a-4d1c-8114-2c9eb2a13dba\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " Oct 30 13:13:21.967405 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.967174 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-workload-certs\") pod \"b8e7baf7-ef26-41d4-9dec-f4987791862d\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " Oct 30 13:13:21.967405 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.967204 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-token\" (UniqueName: \"kubernetes.io/projected/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-token\") pod \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\" (UID: \"41e428c4-45aa-4d5d-b6ea-7939b76b2831\") " Oct 30 13:13:21.967405 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.967245 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8e7baf7-ef26-41d4-9dec-f4987791862d-kube-api-access-k569z\" (UniqueName: \"kubernetes.io/projected/b8e7baf7-ef26-41d4-9dec-f4987791862d-kube-api-access-k569z\") pod \"b8e7baf7-ef26-41d4-9dec-f4987791862d\" (UID: \"b8e7baf7-ef26-41d4-9dec-f4987791862d\") " Oct 30 13:13:21.967405 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.967277 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-workload-socket\") pod \"88155745-668a-4d1c-8114-2c9eb2a13dba\" (UID: \"88155745-668a-4d1c-8114-2c9eb2a13dba\") " Oct 30 13:13:21.967768 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.967745 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-credential-socket" pod "b8e7baf7-ef26-41d4-9dec-f4987791862d" (UID: "b8e7baf7-ef26-41d4-9dec-f4987791862d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.967888 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.967773 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.967975 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.967906 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.967975 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.967924 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.968112 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.967985 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.968112 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.968000 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rpln\" (UniqueName: \"kubernetes.io/projected/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-kube-api-access-4rpln\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.968112 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.968005 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-workload-socket" pod "88155745-668a-4d1c-8114-2c9eb2a13dba" (UID: "88155745-668a-4d1c-8114-2c9eb2a13dba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.968112 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.968019 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.968112 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.968034 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.968112 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.968047 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.968112 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.968062 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.968112 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.968076 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8e76b7-c2ba-4dd2-8b49-2d4565580919-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.968112 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.968090 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8e7baf7-ef26-41d4-9dec-f4987791862d-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:21.968564 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.968347 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-workload-certs" pod "b8e7baf7-ef26-41d4-9dec-f4987791862d" (UID: "b8e7baf7-ef26-41d4-9dec-f4987791862d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.968564 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.968368 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-data" pod "b8e7baf7-ef26-41d4-9dec-f4987791862d" (UID: "b8e7baf7-ef26-41d4-9dec-f4987791862d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.968680 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.968595 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-credential-socket" pod "41e428c4-45aa-4d5d-b6ea-7939b76b2831" (UID: "41e428c4-45aa-4d5d-b6ea-7939b76b2831"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.968743 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.968725 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-data" pod "41e428c4-45aa-4d5d-b6ea-7939b76b2831" (UID: "41e428c4-45aa-4d5d-b6ea-7939b76b2831"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.968918 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.968895 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istiod-ca-cert" pod "41e428c4-45aa-4d5d-b6ea-7939b76b2831" (UID: "41e428c4-45aa-4d5d-b6ea-7939b76b2831"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:21.969718 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.969682 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-workload-certs" pod "41e428c4-45aa-4d5d-b6ea-7939b76b2831" (UID: "41e428c4-45aa-4d5d-b6ea-7939b76b2831"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.969718 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.969694 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-workload-certs" pod "88155745-668a-4d1c-8114-2c9eb2a13dba" (UID: "88155745-668a-4d1c-8114-2c9eb2a13dba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.969887 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.969764 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-envoy" pod "b8e7baf7-ef26-41d4-9dec-f4987791862d" (UID: "b8e7baf7-ef26-41d4-9dec-f4987791862d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.970226 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.970181 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88155745-668a-4d1c-8114-2c9eb2a13dba-istiod-ca-cert" pod "88155745-668a-4d1c-8114-2c9eb2a13dba" (UID: "88155745-668a-4d1c-8114-2c9eb2a13dba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:21.970512 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.970490 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-data" pod "88155745-668a-4d1c-8114-2c9eb2a13dba" (UID: "88155745-668a-4d1c-8114-2c9eb2a13dba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.972218 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.972186 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-credential-socket" pod "88155745-668a-4d1c-8114-2c9eb2a13dba" (UID: "88155745-668a-4d1c-8114-2c9eb2a13dba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.972649 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.972313 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-workload-socket" pod "b8e7baf7-ef26-41d4-9dec-f4987791862d" (UID: "b8e7baf7-ef26-41d4-9dec-f4987791862d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.974563 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.974405 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-envoy" pod "41e428c4-45aa-4d5d-b6ea-7939b76b2831" (UID: "41e428c4-45aa-4d5d-b6ea-7939b76b2831"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.974563 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.974409 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-podinfo" pod "88155745-668a-4d1c-8114-2c9eb2a13dba" (UID: "88155745-668a-4d1c-8114-2c9eb2a13dba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:21.974563 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.974487 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8e7baf7-ef26-41d4-9dec-f4987791862d-kube-api-access-k569z" pod "b8e7baf7-ef26-41d4-9dec-f4987791862d" (UID: "b8e7baf7-ef26-41d4-9dec-f4987791862d"). InnerVolumeSpecName "kube-api-access-k569z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:21.974563 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.974501 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-envoy" pod "88155745-668a-4d1c-8114-2c9eb2a13dba" (UID: "88155745-668a-4d1c-8114-2c9eb2a13dba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:21.974563 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.974510 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-token" pod "41e428c4-45aa-4d5d-b6ea-7939b76b2831" (UID: "41e428c4-45aa-4d5d-b6ea-7939b76b2831"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:21.974563 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.974542 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-podinfo" pod "41e428c4-45aa-4d5d-b6ea-7939b76b2831" (UID: "41e428c4-45aa-4d5d-b6ea-7939b76b2831"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:21.974789 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.974582 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-podinfo" pod "b8e7baf7-ef26-41d4-9dec-f4987791862d" (UID: "b8e7baf7-ef26-41d4-9dec-f4987791862d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:21.975239 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.975217 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-token" pod "88155745-668a-4d1c-8114-2c9eb2a13dba" (UID: "88155745-668a-4d1c-8114-2c9eb2a13dba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:21.975325 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.975310 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-token" pod "b8e7baf7-ef26-41d4-9dec-f4987791862d" (UID: "b8e7baf7-ef26-41d4-9dec-f4987791862d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:21.975404 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.975390 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88155745-668a-4d1c-8114-2c9eb2a13dba-kube-api-access-nlrnh" pod "88155745-668a-4d1c-8114-2c9eb2a13dba" (UID: "88155745-668a-4d1c-8114-2c9eb2a13dba"). InnerVolumeSpecName "kube-api-access-nlrnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:21.976030 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:21.976002 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e428c4-45aa-4d5d-b6ea-7939b76b2831-kube-api-access-xcvnh" pod "41e428c4-45aa-4d5d-b6ea-7939b76b2831" (UID: "41e428c4-45aa-4d5d-b6ea-7939b76b2831"). InnerVolumeSpecName "kube-api-access-xcvnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:22.070043 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.069972 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.070043 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.070011 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.070600 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.070545 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.070841 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.070773 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.070841 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.070799 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.070813 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.071126 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.071140 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.071155 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k569z\" (UniqueName: \"kubernetes.io/projected/b8e7baf7-ef26-41d4-9dec-f4987791862d-kube-api-access-k569z\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.071170 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.071184 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.071199 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.071214 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.071228 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.071242 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.071255 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88155745-668a-4d1c-8114-2c9eb2a13dba-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.071269 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.071282 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.071296 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e428c4-45aa-4d5d-b6ea-7939b76b2831-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.071309 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88155745-668a-4d1c-8114-2c9eb2a13dba-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.071322 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88155745-668a-4d1c-8114-2c9eb2a13dba-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.071335 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcvnh\" (UniqueName: \"kubernetes.io/projected/41e428c4-45aa-4d5d-b6ea-7939b76b2831-kube-api-access-xcvnh\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.071349 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8e7baf7-ef26-41d4-9dec-f4987791862d-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.071366 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlrnh\" (UniqueName: \"kubernetes.io/projected/88155745-668a-4d1c-8114-2c9eb2a13dba-kube-api-access-nlrnh\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.071402 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.071380 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e428c4-45aa-4d5d-b6ea-7939b76b2831-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:22.090960 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.090791 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7" Oct 30 13:13:22.124589 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.123439 4503 generic.go:356] "Generic (PLEG): container finished" podID="88155745-668a-4d1c-8114-2c9eb2a13dba" containerID="a4c7b83daa8ef389d5976f326928a58d0fcbb4c86cffaf6c5bf486d82e71a448" exitCode=0 Oct 30 13:13:22.124589 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.123628 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" Oct 30 13:13:22.124589 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.124051 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" event={"ID":"88155745-668a-4d1c-8114-2c9eb2a13dba","Type":"ContainerDied","Data":"a4c7b83daa8ef389d5976f326928a58d0fcbb4c86cffaf6c5bf486d82e71a448"} Oct 30 13:13:22.124589 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.124115 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29" event={"ID":"88155745-668a-4d1c-8114-2c9eb2a13dba","Type":"ContainerDied","Data":"15d4d353f490ae8b3899d91261d63899c698842eb20c36256a496afa1c72224c"} Oct 30 13:13:22.124589 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.124143 4503 scope.go:122] "RemoveContainer" containerID="a4c7b83daa8ef389d5976f326928a58d0fcbb4c86cffaf6c5bf486d82e71a448" Oct 30 13:13:22.137660 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.136825 4503 generic.go:356] "Generic (PLEG): container finished" podID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" containerID="dbff5914e675f231f498e2360270d32d3712e2debf16db9d305e229a1ce35845" exitCode=0 Oct 30 13:13:22.138189 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.138128 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" Oct 30 13:13:22.147209 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.143366 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" event={"ID":"41e428c4-45aa-4d5d-b6ea-7939b76b2831","Type":"ContainerDied","Data":"dbff5914e675f231f498e2360270d32d3712e2debf16db9d305e229a1ce35845"} Oct 30 13:13:22.147209 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.143426 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g" event={"ID":"41e428c4-45aa-4d5d-b6ea-7939b76b2831","Type":"ContainerDied","Data":"9d81b224a3ec9efa287db71ff5543e019fe799d69ea00728169762621d7da1ab"} Oct 30 13:13:22.150597 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.150122 4503 scope.go:122] "RemoveContainer" containerID="a4c7b83daa8ef389d5976f326928a58d0fcbb4c86cffaf6c5bf486d82e71a448" Oct 30 13:13:22.153577 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:13:22.153414 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c7b83daa8ef389d5976f326928a58d0fcbb4c86cffaf6c5bf486d82e71a448\": container with ID starting with a4c7b83daa8ef389d5976f326928a58d0fcbb4c86cffaf6c5bf486d82e71a448 not found: ID does not exist" containerID="a4c7b83daa8ef389d5976f326928a58d0fcbb4c86cffaf6c5bf486d82e71a448" Oct 30 13:13:22.153577 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.153457 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c7b83daa8ef389d5976f326928a58d0fcbb4c86cffaf6c5bf486d82e71a448"} err="failed to get container status \"a4c7b83daa8ef389d5976f326928a58d0fcbb4c86cffaf6c5bf486d82e71a448\": rpc error: code = NotFound desc = could not find container \"a4c7b83daa8ef389d5976f326928a58d0fcbb4c86cffaf6c5bf486d82e71a448\": container with ID starting with a4c7b83daa8ef389d5976f326928a58d0fcbb4c86cffaf6c5bf486d82e71a448 not found: ID does not exist" Oct 30 13:13:22.153577 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.153481 4503 scope.go:122] "RemoveContainer" containerID="dbff5914e675f231f498e2360270d32d3712e2debf16db9d305e229a1ce35845" Oct 30 13:13:22.154435 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.154406 4503 generic.go:356] "Generic (PLEG): container finished" podID="b8e7baf7-ef26-41d4-9dec-f4987791862d" containerID="0ba68ef1b81e853d282700ae458baef67793bc3b8b558c1f749a9a56885662f0" exitCode=0 Oct 30 13:13:22.155486 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.154961 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" Oct 30 13:13:22.155486 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.155206 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" event={"ID":"b8e7baf7-ef26-41d4-9dec-f4987791862d","Type":"ContainerDied","Data":"0ba68ef1b81e853d282700ae458baef67793bc3b8b558c1f749a9a56885662f0"} Oct 30 13:13:22.155486 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.155252 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw" event={"ID":"b8e7baf7-ef26-41d4-9dec-f4987791862d","Type":"ContainerDied","Data":"0166cedfa35e32f22096a833162dff0c01cc1ec211c6dc25de7f2d8e8eed26f1"} Oct 30 13:13:22.160341 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.160158 4503 generic.go:356] "Generic (PLEG): container finished" podID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerID="950cdbe3dbb332f000742d6b98c34775166e3efde6b0e30622cbb12a2e0eb3eb" exitCode=0 Oct 30 13:13:22.160341 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.160198 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" event={"ID":"2c8e76b7-c2ba-4dd2-8b49-2d4565580919","Type":"ContainerDied","Data":"950cdbe3dbb332f000742d6b98c34775166e3efde6b0e30622cbb12a2e0eb3eb"} Oct 30 13:13:22.160341 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.160240 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" event={"ID":"2c8e76b7-c2ba-4dd2-8b49-2d4565580919","Type":"ContainerDied","Data":"2c924ac93ea0395ca4265e7adfac188715b42b9f3d065c3c0f97ae508c7fc3b7"} Oct 30 13:13:22.160341 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.160316 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb" Oct 30 13:13:22.179568 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.179531 4503 scope.go:122] "RemoveContainer" containerID="dbff5914e675f231f498e2360270d32d3712e2debf16db9d305e229a1ce35845" Oct 30 13:13:22.182702 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:13:22.182640 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbff5914e675f231f498e2360270d32d3712e2debf16db9d305e229a1ce35845\": container with ID starting with dbff5914e675f231f498e2360270d32d3712e2debf16db9d305e229a1ce35845 not found: ID does not exist" containerID="dbff5914e675f231f498e2360270d32d3712e2debf16db9d305e229a1ce35845" Oct 30 13:13:22.182884 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.182853 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbff5914e675f231f498e2360270d32d3712e2debf16db9d305e229a1ce35845"} err="failed to get container status \"dbff5914e675f231f498e2360270d32d3712e2debf16db9d305e229a1ce35845\": rpc error: code = NotFound desc = could not find container \"dbff5914e675f231f498e2360270d32d3712e2debf16db9d305e229a1ce35845\": container with ID starting with dbff5914e675f231f498e2360270d32d3712e2debf16db9d305e229a1ce35845 not found: ID does not exist" Oct 30 13:13:22.184795 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.184776 4503 scope.go:122] "RemoveContainer" containerID="0ba68ef1b81e853d282700ae458baef67793bc3b8b558c1f749a9a56885662f0" Oct 30 13:13:22.211167 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.211125 4503 scope.go:122] "RemoveContainer" containerID="0ba68ef1b81e853d282700ae458baef67793bc3b8b558c1f749a9a56885662f0" Oct 30 13:13:22.211498 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:13:22.211475 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba68ef1b81e853d282700ae458baef67793bc3b8b558c1f749a9a56885662f0\": container with ID starting with 0ba68ef1b81e853d282700ae458baef67793bc3b8b558c1f749a9a56885662f0 not found: ID does not exist" containerID="0ba68ef1b81e853d282700ae458baef67793bc3b8b558c1f749a9a56885662f0" Oct 30 13:13:22.211584 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.211508 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba68ef1b81e853d282700ae458baef67793bc3b8b558c1f749a9a56885662f0"} err="failed to get container status \"0ba68ef1b81e853d282700ae458baef67793bc3b8b558c1f749a9a56885662f0\": rpc error: code = NotFound desc = could not find container \"0ba68ef1b81e853d282700ae458baef67793bc3b8b558c1f749a9a56885662f0\": container with ID starting with 0ba68ef1b81e853d282700ae458baef67793bc3b8b558c1f749a9a56885662f0 not found: ID does not exist" Oct 30 13:13:22.211584 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.211528 4503 scope.go:122] "RemoveContainer" containerID="950cdbe3dbb332f000742d6b98c34775166e3efde6b0e30622cbb12a2e0eb3eb" Oct 30 13:13:22.227069 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.226893 4503 scope.go:122] "RemoveContainer" containerID="950cdbe3dbb332f000742d6b98c34775166e3efde6b0e30622cbb12a2e0eb3eb" Oct 30 13:13:22.229332 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:13:22.229302 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"950cdbe3dbb332f000742d6b98c34775166e3efde6b0e30622cbb12a2e0eb3eb\": container with ID starting with 950cdbe3dbb332f000742d6b98c34775166e3efde6b0e30622cbb12a2e0eb3eb not found: ID does not exist" containerID="950cdbe3dbb332f000742d6b98c34775166e3efde6b0e30622cbb12a2e0eb3eb" Oct 30 13:13:22.229466 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.229341 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"950cdbe3dbb332f000742d6b98c34775166e3efde6b0e30622cbb12a2e0eb3eb"} err="failed to get container status \"950cdbe3dbb332f000742d6b98c34775166e3efde6b0e30622cbb12a2e0eb3eb\": rpc error: code = NotFound desc = could not find container \"950cdbe3dbb332f000742d6b98c34775166e3efde6b0e30622cbb12a2e0eb3eb\": container with ID starting with 950cdbe3dbb332f000742d6b98c34775166e3efde6b0e30622cbb12a2e0eb3eb not found: ID does not exist" Oct 30 13:13:22.237284 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.237262 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29"] Oct 30 13:13:22.247774 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.247745 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-45b29"] Oct 30 13:13:22.283669 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.283636 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g"] Oct 30 13:13:22.295185 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.295163 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-q2g2g"] Oct 30 13:13:22.330831 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.330794 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb"] Oct 30 13:13:22.339661 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.339634 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-n7scb"] Oct 30 13:13:22.361034 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.361007 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" path="/var/lib/kubelet/pods/2c8e76b7-c2ba-4dd2-8b49-2d4565580919/volumes" Oct 30 13:13:22.361508 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.361493 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" path="/var/lib/kubelet/pods/41e428c4-45aa-4d5d-b6ea-7939b76b2831/volumes" Oct 30 13:13:22.361955 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.361930 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88155745-668a-4d1c-8114-2c9eb2a13dba" path="/var/lib/kubelet/pods/88155745-668a-4d1c-8114-2c9eb2a13dba/volumes" Oct 30 13:13:22.403663 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.403629 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw"] Oct 30 13:13:22.407009 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.406970 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb71e2-openshift-default-7c5c5c884c-4wxfw"] Oct 30 13:13:22.433136 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:22.433105 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7"] Oct 30 13:13:22.435425 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:13:22.435393 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d6f01a_f566_41ec_8d49_5dbc377ea951.slice/crio-f24f13de59083b9f43f9d66e598f5cd71904c9fef9ee44c481cba0a55b49d4c2 WatchSource:0}: Error finding container f24f13de59083b9f43f9d66e598f5cd71904c9fef9ee44c481cba0a55b49d4c2: Status 404 returned error can't find the container with id f24f13de59083b9f43f9d66e598f5cd71904c9fef9ee44c481cba0a55b49d4c2 Oct 30 13:13:23.184874 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:23.184830 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7" event={"ID":"c8d6f01a-f566-41ec-8d49-5dbc377ea951","Type":"ContainerStarted","Data":"3a6a88cb2c061356dd5cee21d712e8e9ae965a862e33911621af3d7d4c005447"} Oct 30 13:13:23.185322 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:23.185288 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7" event={"ID":"c8d6f01a-f566-41ec-8d49-5dbc377ea951","Type":"ContainerStarted","Data":"f24f13de59083b9f43f9d66e598f5cd71904c9fef9ee44c481cba0a55b49d4c2"} Oct 30 13:13:23.228427 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:23.228382 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7" podStartSLOduration=2.22836844 podStartE2EDuration="2.22836844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:23.225820447 +0000 UTC m=+1297.049048782" watchObservedRunningTime="2026-10-30 13:13:23.22836844 +0000 UTC m=+1297.051596767" Oct 30 13:13:24.195291 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:24.195256 4503 generic.go:356] "Generic (PLEG): container finished" podID="c8d6f01a-f566-41ec-8d49-5dbc377ea951" containerID="3a6a88cb2c061356dd5cee21d712e8e9ae965a862e33911621af3d7d4c005447" exitCode=0 Oct 30 13:13:24.195739 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:24.195350 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7" event={"ID":"c8d6f01a-f566-41ec-8d49-5dbc377ea951","Type":"ContainerDied","Data":"3a6a88cb2c061356dd5cee21d712e8e9ae965a862e33911621af3d7d4c005447"} Oct 30 13:13:24.362185 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:24.362148 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8e7baf7-ef26-41d4-9dec-f4987791862d" path="/var/lib/kubelet/pods/b8e7baf7-ef26-41d4-9dec-f4987791862d/volumes" Oct 30 13:13:25.341570 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:25.341540 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7" Oct 30 13:13:25.407764 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:25.407731 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8d6f01a-f566-41ec-8d49-5dbc377ea951-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d6f01a-f566-41ec-8d49-5dbc377ea951-maas-api-xks-ca\") pod \"c8d6f01a-f566-41ec-8d49-5dbc377ea951\" (UID: \"c8d6f01a-f566-41ec-8d49-5dbc377ea951\") " Oct 30 13:13:25.407952 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:25.407789 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8d6f01a-f566-41ec-8d49-5dbc377ea951-kube-api-access-9xbwx\" (UniqueName: \"kubernetes.io/projected/c8d6f01a-f566-41ec-8d49-5dbc377ea951-kube-api-access-9xbwx\") pod \"c8d6f01a-f566-41ec-8d49-5dbc377ea951\" (UID: \"c8d6f01a-f566-41ec-8d49-5dbc377ea951\") " Oct 30 13:13:25.407952 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:25.407815 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8d6f01a-f566-41ec-8d49-5dbc377ea951-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d6f01a-f566-41ec-8d49-5dbc377ea951-maas-api-openshift-ca\") pod \"c8d6f01a-f566-41ec-8d49-5dbc377ea951\" (UID: \"c8d6f01a-f566-41ec-8d49-5dbc377ea951\") " Oct 30 13:13:25.408301 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:25.408273 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d6f01a-f566-41ec-8d49-5dbc377ea951-maas-api-openshift-ca" pod "c8d6f01a-f566-41ec-8d49-5dbc377ea951" (UID: "c8d6f01a-f566-41ec-8d49-5dbc377ea951"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:25.409921 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:25.409901 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d6f01a-f566-41ec-8d49-5dbc377ea951-kube-api-access-9xbwx" pod "c8d6f01a-f566-41ec-8d49-5dbc377ea951" (UID: "c8d6f01a-f566-41ec-8d49-5dbc377ea951"). InnerVolumeSpecName "kube-api-access-9xbwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:25.410026 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:25.409987 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8d6f01a-f566-41ec-8d49-5dbc377ea951-maas-api-xks-ca" pod "c8d6f01a-f566-41ec-8d49-5dbc377ea951" (UID: "c8d6f01a-f566-41ec-8d49-5dbc377ea951"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:13:25.508732 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:25.508647 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xbwx\" (UniqueName: \"kubernetes.io/projected/c8d6f01a-f566-41ec-8d49-5dbc377ea951-kube-api-access-9xbwx\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:25.508732 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:25.508680 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d6f01a-f566-41ec-8d49-5dbc377ea951-maas-api-openshift-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:25.508732 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:25.508693 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d6f01a-f566-41ec-8d49-5dbc377ea951-maas-api-xks-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:26.209376 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:26.209347 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7" Oct 30 13:13:26.209575 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:26.209421 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7" event={"ID":"c8d6f01a-f566-41ec-8d49-5dbc377ea951","Type":"ContainerDied","Data":"f24f13de59083b9f43f9d66e598f5cd71904c9fef9ee44c481cba0a55b49d4c2"} Oct 30 13:13:26.209575 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:26.209482 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f24f13de59083b9f43f9d66e598f5cd71904c9fef9ee44c481cba0a55b49d4c2" Oct 30 13:13:26.376998 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:26.376957 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" podUID="d1e386dc-1501-492d-a339-f83aa21f24fe" containerName="payload-processing" containerID="cri-o://14bc85bc358c2ebc6d541ffbd34292dff64c4a2d8241629defbb97468d2efd26" gracePeriod=25 Oct 30 13:13:29.700250 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:13:29.700168 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:13:30.726166 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:30.726126 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" containerID="cri-o://3769555a6057686e0154b6852bb172107515c671fdf7f856c5fa7e87654321de" gracePeriod=45 Oct 30 13:13:30.988506 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:30.988452 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:13:31.062392 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.062357 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-model-cache\") pod \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " Oct 30 13:13:31.062589 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.062418 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-home\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-home\") pod \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " Oct 30 13:13:31.062589 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.062449 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-tls-certs\") pod \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " Oct 30 13:13:31.062589 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.062501 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-tmp-dir\") pod \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " Oct 30 13:13:31.062589 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.062561 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-dshm\") pod \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " Oct 30 13:13:31.062820 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.062613 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-kube-api-access-p54x4\" (UniqueName: \"kubernetes.io/projected/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-kube-api-access-p54x4\") pod \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\" (UID: \"0e3ae60c-3a8a-439f-877e-ea7e8e890b11\") " Oct 30 13:13:31.062971 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.062918 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-home" pod "0e3ae60c-3a8a-439f-877e-ea7e8e890b11" (UID: "0e3ae60c-3a8a-439f-877e-ea7e8e890b11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:31.063110 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.063081 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-model-cache" pod "0e3ae60c-3a8a-439f-877e-ea7e8e890b11" (UID: "0e3ae60c-3a8a-439f-877e-ea7e8e890b11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:31.063175 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.063148 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-tmp-dir" pod "0e3ae60c-3a8a-439f-877e-ea7e8e890b11" (UID: "0e3ae60c-3a8a-439f-877e-ea7e8e890b11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:31.065228 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.065201 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-kube-api-access-p54x4" pod "0e3ae60c-3a8a-439f-877e-ea7e8e890b11" (UID: "0e3ae60c-3a8a-439f-877e-ea7e8e890b11"). InnerVolumeSpecName "kube-api-access-p54x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:31.065323 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.065274 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-tls-certs" pod "0e3ae60c-3a8a-439f-877e-ea7e8e890b11" (UID: "0e3ae60c-3a8a-439f-877e-ea7e8e890b11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:13:31.065367 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.065343 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-dshm" pod "0e3ae60c-3a8a-439f-877e-ea7e8e890b11" (UID: "0e3ae60c-3a8a-439f-877e-ea7e8e890b11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:31.163970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.163903 4503 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-tmp-dir\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:31.163970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.163932 4503 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-dshm\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:31.163970 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.163977 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p54x4\" (UniqueName: \"kubernetes.io/projected/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-kube-api-access-p54x4\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:31.164291 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.163986 4503 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-model-cache\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:31.164291 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.163994 4503 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-home\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:31.164291 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.164002 4503 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3ae60c-3a8a-439f-877e-ea7e8e890b11-tls-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:31.241680 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.241587 4503 generic.go:356] "Generic (PLEG): container finished" podID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerID="3769555a6057686e0154b6852bb172107515c671fdf7f856c5fa7e87654321de" exitCode=0 Oct 30 13:13:31.241680 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.241652 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" event={"ID":"0e3ae60c-3a8a-439f-877e-ea7e8e890b11","Type":"ContainerDied","Data":"3769555a6057686e0154b6852bb172107515c671fdf7f856c5fa7e87654321de"} Oct 30 13:13:31.241883 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.241692 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" Oct 30 13:13:31.241883 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.241711 4503 scope.go:122] "RemoveContainer" containerID="3769555a6057686e0154b6852bb172107515c671fdf7f856c5fa7e87654321de" Oct 30 13:13:31.241883 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.241696 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm" event={"ID":"0e3ae60c-3a8a-439f-877e-ea7e8e890b11","Type":"ContainerDied","Data":"88e7e98a6d52242ba6c8a312494633534c317fb5d25ea8db6f9b9b929b01ce5a"} Oct 30 13:13:31.251983 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.251960 4503 scope.go:122] "RemoveContainer" containerID="3769555a6057686e0154b6852bb172107515c671fdf7f856c5fa7e87654321de" Oct 30 13:13:31.252251 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:13:31.252232 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3769555a6057686e0154b6852bb172107515c671fdf7f856c5fa7e87654321de\": container with ID starting with 3769555a6057686e0154b6852bb172107515c671fdf7f856c5fa7e87654321de not found: ID does not exist" containerID="3769555a6057686e0154b6852bb172107515c671fdf7f856c5fa7e87654321de" Oct 30 13:13:31.252297 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.252261 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3769555a6057686e0154b6852bb172107515c671fdf7f856c5fa7e87654321de"} err="failed to get container status \"3769555a6057686e0154b6852bb172107515c671fdf7f856c5fa7e87654321de\": rpc error: code = NotFound desc = could not find container \"3769555a6057686e0154b6852bb172107515c671fdf7f856c5fa7e87654321de\": container with ID starting with 3769555a6057686e0154b6852bb172107515c671fdf7f856c5fa7e87654321de not found: ID does not exist" Oct 30 13:13:31.295359 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.295323 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm"] Oct 30 13:13:31.300807 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.300781 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-175831/distinct-w5-175831-kserve-d98f98cfc-vvzpm"] Oct 30 13:13:31.626368 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:31.626281 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" podUID="d1e386dc-1501-492d-a339-f83aa21f24fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:13:32.361075 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:32.361034 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" path="/var/lib/kubelet/pods/0e3ae60c-3a8a-439f-877e-ea7e8e890b11/volumes" Oct 30 13:13:36.828122 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:36.826738 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q"] Oct 30 13:13:36.828122 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:36.827277 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" podUID="15a70b19-c9b4-436f-9a48-a45fd8d158e0" containerName="maas-api" containerID="cri-o://1cf6e46021d8a5fee5758048e06390a58bddabab7195f0bb38f3ea8f731ac5a1" gracePeriod=30 Oct 30 13:13:37.139109 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.138786 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" Oct 30 13:13:37.274748 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.274713 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15a70b19-c9b4-436f-9a48-a45fd8d158e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a70b19-c9b4-436f-9a48-a45fd8d158e0-metrics-certs\") pod \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\" (UID: \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\") " Oct 30 13:13:37.275043 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.274787 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15a70b19-c9b4-436f-9a48-a45fd8d158e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a70b19-c9b4-436f-9a48-a45fd8d158e0-openshift-service-ca\") pod \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\" (UID: \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\") " Oct 30 13:13:37.275043 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.274847 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a70b19-c9b4-436f-9a48-a45fd8d158e0-kube-api-access-cdr84\" (UniqueName: \"kubernetes.io/projected/15a70b19-c9b4-436f-9a48-a45fd8d158e0-kube-api-access-cdr84\") pod \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\" (UID: \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\") " Oct 30 13:13:37.275043 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.274902 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15a70b19-c9b4-436f-9a48-a45fd8d158e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a70b19-c9b4-436f-9a48-a45fd8d158e0-maas-api-tls\") pod \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\" (UID: \"15a70b19-c9b4-436f-9a48-a45fd8d158e0\") " Oct 30 13:13:37.275912 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.275883 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a70b19-c9b4-436f-9a48-a45fd8d158e0-openshift-service-ca" pod "15a70b19-c9b4-436f-9a48-a45fd8d158e0" (UID: "15a70b19-c9b4-436f-9a48-a45fd8d158e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:37.290530 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.290476 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a70b19-c9b4-436f-9a48-a45fd8d158e0-maas-api-tls" pod "15a70b19-c9b4-436f-9a48-a45fd8d158e0" (UID: "15a70b19-c9b4-436f-9a48-a45fd8d158e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:13:37.290826 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.290800 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a70b19-c9b4-436f-9a48-a45fd8d158e0-metrics-certs" pod "15a70b19-c9b4-436f-9a48-a45fd8d158e0" (UID: "15a70b19-c9b4-436f-9a48-a45fd8d158e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:13:37.291669 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.291633 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a70b19-c9b4-436f-9a48-a45fd8d158e0-kube-api-access-cdr84" pod "15a70b19-c9b4-436f-9a48-a45fd8d158e0" (UID: "15a70b19-c9b4-436f-9a48-a45fd8d158e0"). InnerVolumeSpecName "kube-api-access-cdr84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:37.309265 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.309224 4503 generic.go:356] "Generic (PLEG): container finished" podID="15a70b19-c9b4-436f-9a48-a45fd8d158e0" containerID="1cf6e46021d8a5fee5758048e06390a58bddabab7195f0bb38f3ea8f731ac5a1" exitCode=0 Oct 30 13:13:37.309405 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.309274 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" Oct 30 13:13:37.309405 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.309306 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" event={"ID":"15a70b19-c9b4-436f-9a48-a45fd8d158e0","Type":"ContainerDied","Data":"1cf6e46021d8a5fee5758048e06390a58bddabab7195f0bb38f3ea8f731ac5a1"} Oct 30 13:13:37.309405 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.309372 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q" event={"ID":"15a70b19-c9b4-436f-9a48-a45fd8d158e0","Type":"ContainerDied","Data":"b7de9d06b5e1fb79e1089c9adc925a49d64d4fe59505b3ffd195650664ab7f33"} Oct 30 13:13:37.309405 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.309401 4503 scope.go:122] "RemoveContainer" containerID="1cf6e46021d8a5fee5758048e06390a58bddabab7195f0bb38f3ea8f731ac5a1" Oct 30 13:13:37.323595 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.323568 4503 scope.go:122] "RemoveContainer" containerID="1cf6e46021d8a5fee5758048e06390a58bddabab7195f0bb38f3ea8f731ac5a1" Oct 30 13:13:37.323978 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:13:37.323919 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cf6e46021d8a5fee5758048e06390a58bddabab7195f0bb38f3ea8f731ac5a1\": container with ID starting with 1cf6e46021d8a5fee5758048e06390a58bddabab7195f0bb38f3ea8f731ac5a1 not found: ID does not exist" containerID="1cf6e46021d8a5fee5758048e06390a58bddabab7195f0bb38f3ea8f731ac5a1" Oct 30 13:13:37.324063 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.323972 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cf6e46021d8a5fee5758048e06390a58bddabab7195f0bb38f3ea8f731ac5a1"} err="failed to get container status \"1cf6e46021d8a5fee5758048e06390a58bddabab7195f0bb38f3ea8f731ac5a1\": rpc error: code = NotFound desc = could not find container \"1cf6e46021d8a5fee5758048e06390a58bddabab7195f0bb38f3ea8f731ac5a1\": container with ID starting with 1cf6e46021d8a5fee5758048e06390a58bddabab7195f0bb38f3ea8f731ac5a1 not found: ID does not exist" Oct 30 13:13:37.364643 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.364568 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q"] Oct 30 13:13:37.369805 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.369778 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-175831-5fc5f6d6d4-h7f8q"] Oct 30 13:13:37.376306 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.376277 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a70b19-c9b4-436f-9a48-a45fd8d158e0-maas-api-tls\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:37.376306 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.376300 4503 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a70b19-c9b4-436f-9a48-a45fd8d158e0-metrics-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:37.376306 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.376309 4503 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a70b19-c9b4-436f-9a48-a45fd8d158e0-openshift-service-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:37.376537 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:37.376318 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdr84\" (UniqueName: \"kubernetes.io/projected/15a70b19-c9b4-436f-9a48-a45fd8d158e0-kube-api-access-cdr84\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:38.361057 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:38.361025 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15a70b19-c9b4-436f-9a48-a45fd8d158e0" path="/var/lib/kubelet/pods/15a70b19-c9b4-436f-9a48-a45fd8d158e0/volumes" Oct 30 13:13:41.626570 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:41.626527 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" podUID="d1e386dc-1501-492d-a339-f83aa21f24fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:13:45.091494 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:45.091456 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj"] Oct 30 13:13:45.091980 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:45.091724 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" podUID="3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" containerName="istio-proxy" containerID="cri-o://f9ae286681bf0522e9e08b38fe907de5a342de5a8a0e59bc8469c9a5a8393f9c" gracePeriod=30 Oct 30 13:13:45.103857 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:45.103827 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd"] Oct 30 13:13:45.104082 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:45.104060 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" containerID="cri-o://773d2cdaa980c929632b794e68d5c8a082faa0bf8b6998edbcd09404014067ad" gracePeriod=30 Oct 30 13:13:45.106426 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:45.106402 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf"] Oct 30 13:13:45.106584 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:45.106564 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" containerID="cri-o://4f7a3c1e074e282189a0ef57f60d443a6333f93e6d727d30137ff638e908f6d7" gracePeriod=30 Oct 30 13:13:45.121641 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:45.121617 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh"] Oct 30 13:13:45.121790 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:45.121772 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" podUID="b0c97856-79d6-4471-bcf8-36b0bc4e5037" containerName="istio-proxy" containerID="cri-o://a042f802ce03ee9599b63629ae52c25e2828c9d58f4dde53567666f82ce0e156" gracePeriod=30 Oct 30 13:13:46.630431 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:46.630406 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" Oct 30 13:13:46.772566 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:46.772524 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e386dc-1501-492d-a339-f83aa21f24fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e386dc-1501-492d-a339-f83aa21f24fe-plugins-config-volume\") pod \"d1e386dc-1501-492d-a339-f83aa21f24fe\" (UID: \"d1e386dc-1501-492d-a339-f83aa21f24fe\") " Oct 30 13:13:46.772762 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:46.772709 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e386dc-1501-492d-a339-f83aa21f24fe-kube-api-access-l46x5\" (UniqueName: \"kubernetes.io/projected/d1e386dc-1501-492d-a339-f83aa21f24fe-kube-api-access-l46x5\") pod \"d1e386dc-1501-492d-a339-f83aa21f24fe\" (UID: \"d1e386dc-1501-492d-a339-f83aa21f24fe\") " Oct 30 13:13:46.772869 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:46.772844 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e386dc-1501-492d-a339-f83aa21f24fe-plugins-config-volume" pod "d1e386dc-1501-492d-a339-f83aa21f24fe" (UID: "d1e386dc-1501-492d-a339-f83aa21f24fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:46.773111 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:46.773092 4503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e386dc-1501-492d-a339-f83aa21f24fe-plugins-config-volume\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:46.774867 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:46.774838 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e386dc-1501-492d-a339-f83aa21f24fe-kube-api-access-l46x5" pod "d1e386dc-1501-492d-a339-f83aa21f24fe" (UID: "d1e386dc-1501-492d-a339-f83aa21f24fe"). InnerVolumeSpecName "kube-api-access-l46x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:46.874286 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:46.874249 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l46x5\" (UniqueName: \"kubernetes.io/projected/d1e386dc-1501-492d-a339-f83aa21f24fe-kube-api-access-l46x5\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:47.387318 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:47.387279 4503 generic.go:356] "Generic (PLEG): container finished" podID="d1e386dc-1501-492d-a339-f83aa21f24fe" containerID="14bc85bc358c2ebc6d541ffbd34292dff64c4a2d8241629defbb97468d2efd26" exitCode=0 Oct 30 13:13:47.387519 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:47.387352 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" event={"ID":"d1e386dc-1501-492d-a339-f83aa21f24fe","Type":"ContainerDied","Data":"14bc85bc358c2ebc6d541ffbd34292dff64c4a2d8241629defbb97468d2efd26"} Oct 30 13:13:47.387519 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:47.387393 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" event={"ID":"d1e386dc-1501-492d-a339-f83aa21f24fe","Type":"ContainerDied","Data":"0b4098b175cdca1dbfcb1665ec4cffc0fb24015191da6820a0a58eeb8010b3b7"} Oct 30 13:13:47.387519 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:47.387413 4503 scope.go:122] "RemoveContainer" containerID="14bc85bc358c2ebc6d541ffbd34292dff64c4a2d8241629defbb97468d2efd26" Oct 30 13:13:47.387519 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:47.387354 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr" Oct 30 13:13:47.397211 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:47.397192 4503 scope.go:122] "RemoveContainer" containerID="14bc85bc358c2ebc6d541ffbd34292dff64c4a2d8241629defbb97468d2efd26" Oct 30 13:13:47.397476 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:13:47.397451 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14bc85bc358c2ebc6d541ffbd34292dff64c4a2d8241629defbb97468d2efd26\": container with ID starting with 14bc85bc358c2ebc6d541ffbd34292dff64c4a2d8241629defbb97468d2efd26 not found: ID does not exist" containerID="14bc85bc358c2ebc6d541ffbd34292dff64c4a2d8241629defbb97468d2efd26" Oct 30 13:13:47.397551 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:47.397490 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14bc85bc358c2ebc6d541ffbd34292dff64c4a2d8241629defbb97468d2efd26"} err="failed to get container status \"14bc85bc358c2ebc6d541ffbd34292dff64c4a2d8241629defbb97468d2efd26\": rpc error: code = NotFound desc = could not find container \"14bc85bc358c2ebc6d541ffbd34292dff64c4a2d8241629defbb97468d2efd26\": container with ID starting with 14bc85bc358c2ebc6d541ffbd34292dff64c4a2d8241629defbb97468d2efd26 not found: ID does not exist" Oct 30 13:13:47.445592 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:47.445554 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr"] Oct 30 13:13:47.457607 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:47.457575 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-175831-5fbbd6995d-49fhr"] Oct 30 13:13:48.361189 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:48.361153 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e386dc-1501-492d-a339-f83aa21f24fe" path="/var/lib/kubelet/pods/d1e386dc-1501-492d-a339-f83aa21f24fe/volumes" Oct 30 13:13:49.868197 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:49.868160 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bd48f45d7-tscg6"] Oct 30 13:13:49.868698 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:49.868382 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-bd48f45d7-tscg6" podUID="30d64a87-af35-4a50-8779-fb9db0b61910" containerName="manager" containerID="cri-o://9885d39b921a2389046f33bcdd8cb83ded4c3ff2bae7dba0e2c86aaefe30e2ba" gracePeriod=30 Oct 30 13:13:50.124470 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.124394 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd48f45d7-tscg6" Oct 30 13:13:50.214931 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.214895 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d64a87-af35-4a50-8779-fb9db0b61910-metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d64a87-af35-4a50-8779-fb9db0b61910-metrics-certs\") pod \"30d64a87-af35-4a50-8779-fb9db0b61910\" (UID: \"30d64a87-af35-4a50-8779-fb9db0b61910\") " Oct 30 13:13:50.215141 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.215064 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d64a87-af35-4a50-8779-fb9db0b61910-kube-api-access-bnrt5\" (UniqueName: \"kubernetes.io/projected/30d64a87-af35-4a50-8779-fb9db0b61910-kube-api-access-bnrt5\") pod \"30d64a87-af35-4a50-8779-fb9db0b61910\" (UID: \"30d64a87-af35-4a50-8779-fb9db0b61910\") " Oct 30 13:13:50.215141 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.215130 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d64a87-af35-4a50-8779-fb9db0b61910-webhook-cert\" (UniqueName: \"kubernetes.io/secret/30d64a87-af35-4a50-8779-fb9db0b61910-webhook-cert\") pod \"30d64a87-af35-4a50-8779-fb9db0b61910\" (UID: \"30d64a87-af35-4a50-8779-fb9db0b61910\") " Oct 30 13:13:50.217693 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.217663 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d64a87-af35-4a50-8779-fb9db0b61910-kube-api-access-bnrt5" pod "30d64a87-af35-4a50-8779-fb9db0b61910" (UID: "30d64a87-af35-4a50-8779-fb9db0b61910"). InnerVolumeSpecName "kube-api-access-bnrt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:50.217810 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.217733 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d64a87-af35-4a50-8779-fb9db0b61910-metrics-certs" pod "30d64a87-af35-4a50-8779-fb9db0b61910" (UID: "30d64a87-af35-4a50-8779-fb9db0b61910"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:13:50.217810 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.217776 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d64a87-af35-4a50-8779-fb9db0b61910-webhook-cert" pod "30d64a87-af35-4a50-8779-fb9db0b61910" (UID: "30d64a87-af35-4a50-8779-fb9db0b61910"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:13:50.317039 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.316970 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnrt5\" (UniqueName: \"kubernetes.io/projected/30d64a87-af35-4a50-8779-fb9db0b61910-kube-api-access-bnrt5\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.317039 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.317002 4503 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30d64a87-af35-4a50-8779-fb9db0b61910-webhook-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.317039 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.317012 4503 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d64a87-af35-4a50-8779-fb9db0b61910-metrics-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.389299 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.389271 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:13:50.405805 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.405775 4503 generic.go:356] "Generic (PLEG): container finished" podID="b0c97856-79d6-4471-bcf8-36b0bc4e5037" containerID="a042f802ce03ee9599b63629ae52c25e2828c9d58f4dde53567666f82ce0e156" exitCode=0 Oct 30 13:13:50.405968 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.405844 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" event={"ID":"b0c97856-79d6-4471-bcf8-36b0bc4e5037","Type":"ContainerDied","Data":"a042f802ce03ee9599b63629ae52c25e2828c9d58f4dde53567666f82ce0e156"} Oct 30 13:13:50.407379 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.407352 4503 generic.go:356] "Generic (PLEG): container finished" podID="30d64a87-af35-4a50-8779-fb9db0b61910" containerID="9885d39b921a2389046f33bcdd8cb83ded4c3ff2bae7dba0e2c86aaefe30e2ba" exitCode=0 Oct 30 13:13:50.407481 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.407400 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd48f45d7-tscg6" Oct 30 13:13:50.407481 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.407436 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd48f45d7-tscg6" event={"ID":"30d64a87-af35-4a50-8779-fb9db0b61910","Type":"ContainerDied","Data":"9885d39b921a2389046f33bcdd8cb83ded4c3ff2bae7dba0e2c86aaefe30e2ba"} Oct 30 13:13:50.407599 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.407488 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd48f45d7-tscg6" event={"ID":"30d64a87-af35-4a50-8779-fb9db0b61910","Type":"ContainerDied","Data":"f2b42a9eaf1940700f62440b260137b53a883c8a8eb9324b46c85f818ce60a6b"} Oct 30 13:13:50.407599 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.407518 4503 scope.go:122] "RemoveContainer" containerID="9885d39b921a2389046f33bcdd8cb83ded4c3ff2bae7dba0e2c86aaefe30e2ba" Oct 30 13:13:50.410217 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.410191 4503 generic.go:356] "Generic (PLEG): container finished" podID="3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" containerID="f9ae286681bf0522e9e08b38fe907de5a342de5a8a0e59bc8469c9a5a8393f9c" exitCode=0 Oct 30 13:13:50.410336 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.410296 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" event={"ID":"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b","Type":"ContainerDied","Data":"f9ae286681bf0522e9e08b38fe907de5a342de5a8a0e59bc8469c9a5a8393f9c"} Oct 30 13:13:50.413105 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.412713 4503 generic.go:356] "Generic (PLEG): container finished" podID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerID="773d2cdaa980c929632b794e68d5c8a082faa0bf8b6998edbcd09404014067ad" exitCode=0 Oct 30 13:13:50.413105 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.412811 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" event={"ID":"cd5b17d2-dd01-422f-ade1-6b34aebe0048","Type":"ContainerDied","Data":"773d2cdaa980c929632b794e68d5c8a082faa0bf8b6998edbcd09404014067ad"} Oct 30 13:13:50.415094 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.415068 4503 generic.go:356] "Generic (PLEG): container finished" podID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerID="4f7a3c1e074e282189a0ef57f60d443a6333f93e6d727d30137ff638e908f6d7" exitCode=0 Oct 30 13:13:50.415411 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.415105 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" event={"ID":"79b4bb1d-abcd-4bda-98ed-e62781d1abd7","Type":"ContainerDied","Data":"4f7a3c1e074e282189a0ef57f60d443a6333f93e6d727d30137ff638e908f6d7"} Oct 30 13:13:50.415411 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.415144 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" event={"ID":"79b4bb1d-abcd-4bda-98ed-e62781d1abd7","Type":"ContainerDied","Data":"d6a2ba4d9ae7f214d20da67d34ea1afef0c3e4edf5b1c47a5061bab6a9d65923"} Oct 30 13:13:50.415411 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.415217 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf" Oct 30 13:13:50.417456 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.417436 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-podinfo\") pod \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " Oct 30 13:13:50.419155 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.419131 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-token\" (UniqueName: \"kubernetes.io/projected/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-token\") pod \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " Oct 30 13:13:50.419264 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.419178 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istiod-ca-cert\") pod \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " Oct 30 13:13:50.419264 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.419227 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-envoy\") pod \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " Oct 30 13:13:50.419387 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.419322 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-kube-api-access-njns9\" (UniqueName: \"kubernetes.io/projected/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-kube-api-access-njns9\") pod \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " Oct 30 13:13:50.419446 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.419401 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-workload-certs\") pod \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " Oct 30 13:13:50.419497 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.419445 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-credential-socket\") pod \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " Oct 30 13:13:50.419557 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.419499 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-data\") pod \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " Oct 30 13:13:50.419557 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.419539 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-workload-socket\") pod \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\" (UID: \"79b4bb1d-abcd-4bda-98ed-e62781d1abd7\") " Oct 30 13:13:50.420532 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.420221 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-credential-socket" pod "79b4bb1d-abcd-4bda-98ed-e62781d1abd7" (UID: "79b4bb1d-abcd-4bda-98ed-e62781d1abd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.420532 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.420341 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-workload-certs" pod "79b4bb1d-abcd-4bda-98ed-e62781d1abd7" (UID: "79b4bb1d-abcd-4bda-98ed-e62781d1abd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.420532 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.420428 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istiod-ca-cert" pod "79b4bb1d-abcd-4bda-98ed-e62781d1abd7" (UID: "79b4bb1d-abcd-4bda-98ed-e62781d1abd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:50.420796 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.420748 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-workload-socket" pod "79b4bb1d-abcd-4bda-98ed-e62781d1abd7" (UID: "79b4bb1d-abcd-4bda-98ed-e62781d1abd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.420960 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.420919 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-data" pod "79b4bb1d-abcd-4bda-98ed-e62781d1abd7" (UID: "79b4bb1d-abcd-4bda-98ed-e62781d1abd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.424742 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.424639 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-kube-api-access-njns9" pod "79b4bb1d-abcd-4bda-98ed-e62781d1abd7" (UID: "79b4bb1d-abcd-4bda-98ed-e62781d1abd7"). InnerVolumeSpecName "kube-api-access-njns9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:50.424742 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.424710 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-podinfo" pod "79b4bb1d-abcd-4bda-98ed-e62781d1abd7" (UID: "79b4bb1d-abcd-4bda-98ed-e62781d1abd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:50.424742 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.424733 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-token" pod "79b4bb1d-abcd-4bda-98ed-e62781d1abd7" (UID: "79b4bb1d-abcd-4bda-98ed-e62781d1abd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:50.426093 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.426067 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-envoy" pod "79b4bb1d-abcd-4bda-98ed-e62781d1abd7" (UID: "79b4bb1d-abcd-4bda-98ed-e62781d1abd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.426838 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.426822 4503 scope.go:122] "RemoveContainer" containerID="9885d39b921a2389046f33bcdd8cb83ded4c3ff2bae7dba0e2c86aaefe30e2ba" Oct 30 13:13:50.427206 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:13:50.427162 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9885d39b921a2389046f33bcdd8cb83ded4c3ff2bae7dba0e2c86aaefe30e2ba\": container with ID starting with 9885d39b921a2389046f33bcdd8cb83ded4c3ff2bae7dba0e2c86aaefe30e2ba not found: ID does not exist" containerID="9885d39b921a2389046f33bcdd8cb83ded4c3ff2bae7dba0e2c86aaefe30e2ba" Oct 30 13:13:50.427323 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.427203 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9885d39b921a2389046f33bcdd8cb83ded4c3ff2bae7dba0e2c86aaefe30e2ba"} err="failed to get container status \"9885d39b921a2389046f33bcdd8cb83ded4c3ff2bae7dba0e2c86aaefe30e2ba\": rpc error: code = NotFound desc = could not find container \"9885d39b921a2389046f33bcdd8cb83ded4c3ff2bae7dba0e2c86aaefe30e2ba\": container with ID starting with 9885d39b921a2389046f33bcdd8cb83ded4c3ff2bae7dba0e2c86aaefe30e2ba not found: ID does not exist" Oct 30 13:13:50.427323 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.427229 4503 scope.go:122] "RemoveContainer" containerID="4f7a3c1e074e282189a0ef57f60d443a6333f93e6d727d30137ff638e908f6d7" Oct 30 13:13:50.437003 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.436369 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:13:50.441150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.441122 4503 scope.go:122] "RemoveContainer" containerID="4f7a3c1e074e282189a0ef57f60d443a6333f93e6d727d30137ff638e908f6d7" Oct 30 13:13:50.441569 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:13:50.441511 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7a3c1e074e282189a0ef57f60d443a6333f93e6d727d30137ff638e908f6d7\": container with ID starting with 4f7a3c1e074e282189a0ef57f60d443a6333f93e6d727d30137ff638e908f6d7 not found: ID does not exist" containerID="4f7a3c1e074e282189a0ef57f60d443a6333f93e6d727d30137ff638e908f6d7" Oct 30 13:13:50.441569 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.441543 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7a3c1e074e282189a0ef57f60d443a6333f93e6d727d30137ff638e908f6d7"} err="failed to get container status \"4f7a3c1e074e282189a0ef57f60d443a6333f93e6d727d30137ff638e908f6d7\": rpc error: code = NotFound desc = could not find container \"4f7a3c1e074e282189a0ef57f60d443a6333f93e6d727d30137ff638e908f6d7\": container with ID starting with 4f7a3c1e074e282189a0ef57f60d443a6333f93e6d727d30137ff638e908f6d7 not found: ID does not exist" Oct 30 13:13:50.495027 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.494966 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bd48f45d7-tscg6"] Oct 30 13:13:50.495174 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.495149 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:13:50.501439 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.501411 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bd48f45d7-tscg6"] Oct 30 13:13:50.507420 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.507393 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:13:50.521264 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521238 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-credential-socket\") pod \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " Oct 30 13:13:50.521380 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521274 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-workload-certs\") pod \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " Oct 30 13:13:50.521380 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521306 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-envoy\") pod \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " Oct 30 13:13:50.521380 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521352 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-envoy\") pod \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " Oct 30 13:13:50.521546 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521397 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istiod-ca-cert\") pod \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " Oct 30 13:13:50.521546 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521426 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-credential-socket\") pod \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " Oct 30 13:13:50.521546 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521454 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-podinfo\") pod \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " Oct 30 13:13:50.521546 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521484 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-data\") pod \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " Oct 30 13:13:50.521546 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521487 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-credential-socket" pod "cd5b17d2-dd01-422f-ade1-6b34aebe0048" (UID: "cd5b17d2-dd01-422f-ade1-6b34aebe0048"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.521546 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521509 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-workload-socket\") pod \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " Oct 30 13:13:50.521835 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521563 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-token\" (UniqueName: \"kubernetes.io/projected/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-token\") pod \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " Oct 30 13:13:50.521835 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521568 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-workload-certs" pod "3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" (UID: "3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.521835 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521593 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-data\") pod \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " Oct 30 13:13:50.521835 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521622 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd5b17d2-dd01-422f-ade1-6b34aebe0048-kube-api-access-nmkdx\" (UniqueName: \"kubernetes.io/projected/cd5b17d2-dd01-422f-ade1-6b34aebe0048-kube-api-access-nmkdx\") pod \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " Oct 30 13:13:50.521835 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521642 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c97856-79d6-4471-bcf8-36b0bc4e5037-kube-api-access-46pc6\" (UniqueName: \"kubernetes.io/projected/b0c97856-79d6-4471-bcf8-36b0bc4e5037-kube-api-access-46pc6\") pod \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " Oct 30 13:13:50.521835 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521674 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-workload-socket\") pod \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " Oct 30 13:13:50.521835 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521699 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-kube-api-access-ks98m\" (UniqueName: \"kubernetes.io/projected/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-kube-api-access-ks98m\") pod \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " Oct 30 13:13:50.521835 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521711 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-workload-socket" pod "cd5b17d2-dd01-422f-ade1-6b34aebe0048" (UID: "cd5b17d2-dd01-422f-ade1-6b34aebe0048"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.521835 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521725 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-workload-certs\") pod \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " Oct 30 13:13:50.521835 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521753 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-podinfo\") pod \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " Oct 30 13:13:50.521835 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521788 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istiod-ca-cert\") pod \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " Oct 30 13:13:50.521835 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521828 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-credential-socket" pod "3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" (UID: "3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.521835 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521840 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istiod-ca-cert\") pod \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " Oct 30 13:13:50.522502 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521866 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-podinfo\") pod \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " Oct 30 13:13:50.522502 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521909 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-workload-certs\") pod \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " Oct 30 13:13:50.522502 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.521969 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-envoy\") pod \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " Oct 30 13:13:50.522502 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522002 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-token\" (UniqueName: \"kubernetes.io/projected/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-token\") pod \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " Oct 30 13:13:50.522502 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522039 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-workload-socket\") pod \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\" (UID: \"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b\") " Oct 30 13:13:50.522502 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522067 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-credential-socket\") pod \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " Oct 30 13:13:50.522502 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522097 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-data\") pod \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\" (UID: \"cd5b17d2-dd01-422f-ade1-6b34aebe0048\") " Oct 30 13:13:50.522502 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522123 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-token\" (UniqueName: \"kubernetes.io/projected/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-token\") pod \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\" (UID: \"b0c97856-79d6-4471-bcf8-36b0bc4e5037\") " Oct 30 13:13:50.522502 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522241 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-data" pod "b0c97856-79d6-4471-bcf8-36b0bc4e5037" (UID: "b0c97856-79d6-4471-bcf8-36b0bc4e5037"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.522502 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522249 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-data" pod "3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" (UID: "3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.522502 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522168 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istiod-ca-cert" pod "cd5b17d2-dd01-422f-ade1-6b34aebe0048" (UID: "cd5b17d2-dd01-422f-ade1-6b34aebe0048"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:50.523076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522585 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njns9\" (UniqueName: \"kubernetes.io/projected/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-kube-api-access-njns9\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.523076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522608 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.523076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522623 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.523076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522635 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-workload-certs" pod "cd5b17d2-dd01-422f-ade1-6b34aebe0048" (UID: "cd5b17d2-dd01-422f-ade1-6b34aebe0048"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.523076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522658 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.523076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522674 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.523076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522688 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.523076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522703 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.523076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522717 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.523076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522732 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.523076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522746 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.523076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522758 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.523076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522772 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.523076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522786 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.523076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522798 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b4bb1d-abcd-4bda-98ed-e62781d1abd7-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.523076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522811 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.523076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.522826 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.523076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.523042 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-workload-certs" pod "b0c97856-79d6-4471-bcf8-36b0bc4e5037" (UID: "b0c97856-79d6-4471-bcf8-36b0bc4e5037"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.523724 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.523212 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-workload-socket" pod "b0c97856-79d6-4471-bcf8-36b0bc4e5037" (UID: "b0c97856-79d6-4471-bcf8-36b0bc4e5037"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.523784 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.523743 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istiod-ca-cert" pod "b0c97856-79d6-4471-bcf8-36b0bc4e5037" (UID: "b0c97856-79d6-4471-bcf8-36b0bc4e5037"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:50.524469 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.523952 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-workload-socket" pod "3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" (UID: "3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.524469 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.524101 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-credential-socket" pod "b0c97856-79d6-4471-bcf8-36b0bc4e5037" (UID: "b0c97856-79d6-4471-bcf8-36b0bc4e5037"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.524469 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.524218 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-data" pod "cd5b17d2-dd01-422f-ade1-6b34aebe0048" (UID: "cd5b17d2-dd01-422f-ade1-6b34aebe0048"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.524469 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.524438 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istiod-ca-cert" pod "3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" (UID: "3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:50.525008 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.524980 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-envoy" pod "3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" (UID: "3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.526195 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.526167 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-podinfo" pod "3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" (UID: "3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:50.526321 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.526302 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-envoy" pod "b0c97856-79d6-4471-bcf8-36b0bc4e5037" (UID: "b0c97856-79d6-4471-bcf8-36b0bc4e5037"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.526772 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.526739 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-envoy" pod "cd5b17d2-dd01-422f-ade1-6b34aebe0048" (UID: "cd5b17d2-dd01-422f-ade1-6b34aebe0048"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:50.526772 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.526748 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-token" pod "b0c97856-79d6-4471-bcf8-36b0bc4e5037" (UID: "b0c97856-79d6-4471-bcf8-36b0bc4e5037"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:50.526906 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.526840 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-kube-api-access-ks98m" pod "3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" (UID: "3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b"). InnerVolumeSpecName "kube-api-access-ks98m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:50.527013 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.526904 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd5b17d2-dd01-422f-ade1-6b34aebe0048-kube-api-access-nmkdx" pod "cd5b17d2-dd01-422f-ade1-6b34aebe0048" (UID: "cd5b17d2-dd01-422f-ade1-6b34aebe0048"). InnerVolumeSpecName "kube-api-access-nmkdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:50.527074 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.527007 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-token" pod "cd5b17d2-dd01-422f-ade1-6b34aebe0048" (UID: "cd5b17d2-dd01-422f-ade1-6b34aebe0048"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:50.527111 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.527095 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-podinfo" pod "b0c97856-79d6-4471-bcf8-36b0bc4e5037" (UID: "b0c97856-79d6-4471-bcf8-36b0bc4e5037"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:50.527977 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.527953 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-token" pod "3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" (UID: "3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:50.528154 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.528138 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-podinfo" pod "cd5b17d2-dd01-422f-ade1-6b34aebe0048" (UID: "cd5b17d2-dd01-422f-ade1-6b34aebe0048"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:50.528670 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.528646 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c97856-79d6-4471-bcf8-36b0bc4e5037-kube-api-access-46pc6" pod "b0c97856-79d6-4471-bcf8-36b0bc4e5037" (UID: "b0c97856-79d6-4471-bcf8-36b0bc4e5037"). InnerVolumeSpecName "kube-api-access-46pc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:50.623676 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623644 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.623676 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623669 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.623676 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623679 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmkdx\" (UniqueName: \"kubernetes.io/projected/cd5b17d2-dd01-422f-ade1-6b34aebe0048-kube-api-access-nmkdx\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.623915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623689 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46pc6\" (UniqueName: \"kubernetes.io/projected/b0c97856-79d6-4471-bcf8-36b0bc4e5037-kube-api-access-46pc6\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.623915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623698 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.623915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623706 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks98m\" (UniqueName: \"kubernetes.io/projected/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-kube-api-access-ks98m\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.623915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623715 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.623915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623723 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.623915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623731 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.623915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623738 4503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istiod-ca-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.623915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623745 4503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-podinfo\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.623915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623754 4503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-workload-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.623915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623762 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.623915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623769 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.623915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623778 4503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b-workload-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.623915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623786 4503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-credential-socket\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.623915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623793 4503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-data\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.623915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623801 4503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-token\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.623915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623808 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd5b17d2-dd01-422f-ade1-6b34aebe0048-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.623915 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.623817 4503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c97856-79d6-4471-bcf8-36b0bc4e5037-istio-envoy\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:13:50.763436 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.763407 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf"] Oct 30 13:13:50.769202 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:50.769175 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-b2ccf"] Oct 30 13:13:51.153511 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.153479 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bd48f45d7-zp6j6"] Oct 30 13:13:51.154070 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154052 4503 cpu_manager.go:414] "containerMap: removing container" podUID="d1e386dc-1501-492d-a339-f83aa21f24fe" containerName="payload-processing" Oct 30 13:13:51.154127 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154072 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e386dc-1501-492d-a339-f83aa21f24fe" containerName="payload-processing" Oct 30 13:13:51.154127 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154088 4503 cpu_manager.go:414] "containerMap: removing container" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerName="istio-proxy" Oct 30 13:13:51.154127 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154097 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerName="istio-proxy" Oct 30 13:13:51.154127 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154110 4503 cpu_manager.go:414] "containerMap: removing container" podUID="15a70b19-c9b4-436f-9a48-a45fd8d158e0" containerName="maas-api" Oct 30 13:13:51.154127 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154119 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a70b19-c9b4-436f-9a48-a45fd8d158e0" containerName="maas-api" Oct 30 13:13:51.154308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154133 4503 cpu_manager.go:414] "containerMap: removing container" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" Oct 30 13:13:51.154308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154140 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" Oct 30 13:13:51.154308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154155 4503 cpu_manager.go:414] "containerMap: removing container" podUID="b0c97856-79d6-4471-bcf8-36b0bc4e5037" containerName="istio-proxy" Oct 30 13:13:51.154308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154162 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c97856-79d6-4471-bcf8-36b0bc4e5037" containerName="istio-proxy" Oct 30 13:13:51.154308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154174 4503 cpu_manager.go:414] "containerMap: removing container" podUID="b8e7baf7-ef26-41d4-9dec-f4987791862d" containerName="istio-proxy" Oct 30 13:13:51.154308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154181 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8e7baf7-ef26-41d4-9dec-f4987791862d" containerName="istio-proxy" Oct 30 13:13:51.154308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154204 4503 cpu_manager.go:414] "containerMap: removing container" podUID="c8d6f01a-f566-41ec-8d49-5dbc377ea951" containerName="revoke-keys" Oct 30 13:13:51.154308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154211 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d6f01a-f566-41ec-8d49-5dbc377ea951" containerName="revoke-keys" Oct 30 13:13:51.154308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154223 4503 cpu_manager.go:414] "containerMap: removing container" podUID="88155745-668a-4d1c-8114-2c9eb2a13dba" containerName="istio-proxy" Oct 30 13:13:51.154308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154230 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88155745-668a-4d1c-8114-2c9eb2a13dba" containerName="istio-proxy" Oct 30 13:13:51.154308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154242 4503 cpu_manager.go:414] "containerMap: removing container" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" Oct 30 13:13:51.154308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154248 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" Oct 30 13:13:51.154308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154258 4503 cpu_manager.go:414] "containerMap: removing container" podUID="3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" containerName="istio-proxy" Oct 30 13:13:51.154308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154264 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" containerName="istio-proxy" Oct 30 13:13:51.154308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154277 4503 cpu_manager.go:414] "containerMap: removing container" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" Oct 30 13:13:51.154308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154283 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" Oct 30 13:13:51.154308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154295 4503 cpu_manager.go:414] "containerMap: removing container" podUID="30d64a87-af35-4a50-8779-fb9db0b61910" containerName="manager" Oct 30 13:13:51.154308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154302 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d64a87-af35-4a50-8779-fb9db0b61910" containerName="manager" Oct 30 13:13:51.154882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154318 4503 cpu_manager.go:414] "containerMap: removing container" podUID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" containerName="istio-proxy" Oct 30 13:13:51.154882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154325 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" containerName="istio-proxy" Oct 30 13:13:51.154882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154425 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e386dc-1501-492d-a339-f83aa21f24fe" containerName="payload-processing" Oct 30 13:13:51.154882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154438 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8e76b7-c2ba-4dd2-8b49-2d4565580919" containerName="istio-proxy" Oct 30 13:13:51.154882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154447 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a70b19-c9b4-436f-9a48-a45fd8d158e0" containerName="maas-api" Oct 30 13:13:51.154882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154456 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e3ae60c-3a8a-439f-877e-ea7e8e890b11" containerName="main" Oct 30 13:13:51.154882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154465 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c97856-79d6-4471-bcf8-36b0bc4e5037" containerName="istio-proxy" Oct 30 13:13:51.154882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154474 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8e7baf7-ef26-41d4-9dec-f4987791862d" containerName="istio-proxy" Oct 30 13:13:51.154882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154485 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d6f01a-f566-41ec-8d49-5dbc377ea951" containerName="revoke-keys" Oct 30 13:13:51.154882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154493 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="88155745-668a-4d1c-8114-2c9eb2a13dba" containerName="istio-proxy" Oct 30 13:13:51.154882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154500 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" containerName="istio-proxy" Oct 30 13:13:51.154882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154508 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" containerName="istio-proxy" Oct 30 13:13:51.154882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154517 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" containerName="istio-proxy" Oct 30 13:13:51.154882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154541 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d64a87-af35-4a50-8779-fb9db0b61910" containerName="manager" Oct 30 13:13:51.154882 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.154550 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e428c4-45aa-4d5d-b6ea-7939b76b2831" containerName="istio-proxy" Oct 30 13:13:51.159596 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.159574 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd48f45d7-zp6j6" Oct 30 13:13:51.185599 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.185570 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bd48f45d7-zp6j6"] Oct 30 13:13:51.185895 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.185877 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 13:13:51.186016 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.186003 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 13:13:51.186111 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.186097 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-28684\"" Oct 30 13:13:51.186335 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.186247 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 13:13:51.229800 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.229765 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv4d4\" (UniqueName: \"kubernetes.io/projected/8a39f600-f319-48ff-a1b1-37c523ccc255-kube-api-access-wv4d4\") pod \"maas-controller-bd48f45d7-zp6j6\" (UID: \"8a39f600-f319-48ff-a1b1-37c523ccc255\") " pod="opendatahub/maas-controller-bd48f45d7-zp6j6" Oct 30 13:13:51.229980 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.229832 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a39f600-f319-48ff-a1b1-37c523ccc255-metrics-certs\") pod \"maas-controller-bd48f45d7-zp6j6\" (UID: \"8a39f600-f319-48ff-a1b1-37c523ccc255\") " pod="opendatahub/maas-controller-bd48f45d7-zp6j6" Oct 30 13:13:51.230043 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.229977 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a39f600-f319-48ff-a1b1-37c523ccc255-webhook-cert\") pod \"maas-controller-bd48f45d7-zp6j6\" (UID: \"8a39f600-f319-48ff-a1b1-37c523ccc255\") " pod="opendatahub/maas-controller-bd48f45d7-zp6j6" Oct 30 13:13:51.330588 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.330548 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a39f600-f319-48ff-a1b1-37c523ccc255-webhook-cert\") pod \"maas-controller-bd48f45d7-zp6j6\" (UID: \"8a39f600-f319-48ff-a1b1-37c523ccc255\") " pod="opendatahub/maas-controller-bd48f45d7-zp6j6" Oct 30 13:13:51.330794 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.330632 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv4d4\" (UniqueName: \"kubernetes.io/projected/8a39f600-f319-48ff-a1b1-37c523ccc255-kube-api-access-wv4d4\") pod \"maas-controller-bd48f45d7-zp6j6\" (UID: \"8a39f600-f319-48ff-a1b1-37c523ccc255\") " pod="opendatahub/maas-controller-bd48f45d7-zp6j6" Oct 30 13:13:51.330794 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.330656 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a39f600-f319-48ff-a1b1-37c523ccc255-metrics-certs\") pod \"maas-controller-bd48f45d7-zp6j6\" (UID: \"8a39f600-f319-48ff-a1b1-37c523ccc255\") " pod="opendatahub/maas-controller-bd48f45d7-zp6j6" Oct 30 13:13:51.333212 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.333182 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a39f600-f319-48ff-a1b1-37c523ccc255-metrics-certs\") pod \"maas-controller-bd48f45d7-zp6j6\" (UID: \"8a39f600-f319-48ff-a1b1-37c523ccc255\") " pod="opendatahub/maas-controller-bd48f45d7-zp6j6" Oct 30 13:13:51.333327 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.333232 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a39f600-f319-48ff-a1b1-37c523ccc255-webhook-cert\") pod \"maas-controller-bd48f45d7-zp6j6\" (UID: \"8a39f600-f319-48ff-a1b1-37c523ccc255\") " pod="opendatahub/maas-controller-bd48f45d7-zp6j6" Oct 30 13:13:51.355005 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.354977 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv4d4\" (UniqueName: \"kubernetes.io/projected/8a39f600-f319-48ff-a1b1-37c523ccc255-kube-api-access-wv4d4\") pod \"maas-controller-bd48f45d7-zp6j6\" (UID: \"8a39f600-f319-48ff-a1b1-37c523ccc255\") " pod="opendatahub/maas-controller-bd48f45d7-zp6j6" Oct 30 13:13:51.421667 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.421584 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" Oct 30 13:13:51.421833 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.421584 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd" event={"ID":"cd5b17d2-dd01-422f-ade1-6b34aebe0048","Type":"ContainerDied","Data":"b847034b4f2507c474d7a0357fee909a513e012c69a16b73bb1e47eb926402e1"} Oct 30 13:13:51.421833 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.421735 4503 scope.go:122] "RemoveContainer" containerID="773d2cdaa980c929632b794e68d5c8a082faa0bf8b6998edbcd09404014067ad" Oct 30 13:13:51.424084 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.424039 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" event={"ID":"b0c97856-79d6-4471-bcf8-36b0bc4e5037","Type":"ContainerDied","Data":"2d3698a8256b04e1869b6e2e59ff2df48febad7b47c0686964becf275925f931"} Oct 30 13:13:51.424222 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.424069 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh" Oct 30 13:13:51.426460 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.426440 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" event={"ID":"3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b","Type":"ContainerDied","Data":"ef895a030e1fea22a7e1be93e8939f7e0be07de2d3f1b36f0235ebe62f077076"} Oct 30 13:13:51.426575 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.426481 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj" Oct 30 13:13:51.433912 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.433894 4503 scope.go:122] "RemoveContainer" containerID="a042f802ce03ee9599b63629ae52c25e2828c9d58f4dde53567666f82ce0e156" Oct 30 13:13:51.445141 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.445060 4503 scope.go:122] "RemoveContainer" containerID="f9ae286681bf0522e9e08b38fe907de5a342de5a8a0e59bc8469c9a5a8393f9c" Oct 30 13:13:51.473830 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.473800 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd48f45d7-zp6j6" Oct 30 13:13:51.492964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.492924 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd"] Oct 30 13:13:51.497540 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.497516 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-hstwd"] Oct 30 13:13:51.537933 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.537867 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh"] Oct 30 13:13:51.541637 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.541601 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-xnmxh"] Oct 30 13:13:51.580847 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.580816 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj"] Oct 30 13:13:51.584307 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.584282 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-175831-openshift-default-686b69df8f-kkmmj"] Oct 30 13:13:51.784805 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:51.784781 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bd48f45d7-zp6j6"] Oct 30 13:13:51.786790 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:13:51.786763 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a39f600_f319_48ff_a1b1_37c523ccc255.slice/crio-5bdd5dff217c8bca836c2bcb241e7305934fe3ee364b8197e9f27cc9939f8673 WatchSource:0}: Error finding container 5bdd5dff217c8bca836c2bcb241e7305934fe3ee364b8197e9f27cc9939f8673: Status 404 returned error can't find the container with id 5bdd5dff217c8bca836c2bcb241e7305934fe3ee364b8197e9f27cc9939f8673 Oct 30 13:13:52.363826 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:52.363794 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d64a87-af35-4a50-8779-fb9db0b61910" path="/var/lib/kubelet/pods/30d64a87-af35-4a50-8779-fb9db0b61910/volumes" Oct 30 13:13:52.364344 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:52.364327 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b" path="/var/lib/kubelet/pods/3c024b5b-a00f-4c31-bd17-b9fb2ee58b0b/volumes" Oct 30 13:13:52.364770 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:52.364755 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b4bb1d-abcd-4bda-98ed-e62781d1abd7" path="/var/lib/kubelet/pods/79b4bb1d-abcd-4bda-98ed-e62781d1abd7/volumes" Oct 30 13:13:52.365538 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:52.365517 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c97856-79d6-4471-bcf8-36b0bc4e5037" path="/var/lib/kubelet/pods/b0c97856-79d6-4471-bcf8-36b0bc4e5037/volumes" Oct 30 13:13:52.365987 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:52.365971 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd5b17d2-dd01-422f-ade1-6b34aebe0048" path="/var/lib/kubelet/pods/cd5b17d2-dd01-422f-ade1-6b34aebe0048/volumes" Oct 30 13:13:52.436253 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:52.436213 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd48f45d7-zp6j6" event={"ID":"8a39f600-f319-48ff-a1b1-37c523ccc255","Type":"ContainerStarted","Data":"6a753b9fe1b1432ae9193e095584668fc7d582efd7a171cbecf021912d83ec91"} Oct 30 13:13:52.436437 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:52.436420 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bd48f45d7-zp6j6" Oct 30 13:13:52.436495 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:52.436458 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd48f45d7-zp6j6" event={"ID":"8a39f600-f319-48ff-a1b1-37c523ccc255","Type":"ContainerStarted","Data":"5bdd5dff217c8bca836c2bcb241e7305934fe3ee364b8197e9f27cc9939f8673"} Oct 30 13:13:52.519317 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:13:52.519260 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-bd48f45d7-zp6j6" podStartSLOduration=1.000867962 podStartE2EDuration="1.519245426s" totalImagesPullingTime="518.377464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:52.517544614 +0000 UTC m=+1326.340772895" watchObservedRunningTime="2026-10-30 13:13:52.519245426 +0000 UTC m=+1326.342473702" Oct 30 13:13:59.700888 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:13:59.700846 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:14:03.446398 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:03.446367 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bd48f45d7-zp6j6" Oct 30 13:14:11.328797 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:11.328756 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm"] Oct 30 13:14:11.332348 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:11.332292 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm" Oct 30 13:14:11.366319 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:11.366285 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm"] Oct 30 13:14:11.375599 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:11.375570 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:14:11.375771 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:11.375739 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:14:11.428060 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:11.428023 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tzl7\" (UniqueName: \"kubernetes.io/projected/a2b4f45d-b484-48d6-b6a6-43739a937815-kube-api-access-9tzl7\") pod \"maas-api-revoke-sub-85237e45-t7sjm\" (UID: \"a2b4f45d-b484-48d6-b6a6-43739a937815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm" Oct 30 13:14:11.428257 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:11.428132 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2b4f45d-b484-48d6-b6a6-43739a937815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-t7sjm\" (UID: \"a2b4f45d-b484-48d6-b6a6-43739a937815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm" Oct 30 13:14:11.428257 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:11.428211 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2b4f45d-b484-48d6-b6a6-43739a937815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-t7sjm\" (UID: \"a2b4f45d-b484-48d6-b6a6-43739a937815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm" Oct 30 13:14:11.529200 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:11.529156 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tzl7\" (UniqueName: \"kubernetes.io/projected/a2b4f45d-b484-48d6-b6a6-43739a937815-kube-api-access-9tzl7\") pod \"maas-api-revoke-sub-85237e45-t7sjm\" (UID: \"a2b4f45d-b484-48d6-b6a6-43739a937815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm" Oct 30 13:14:11.529416 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:11.529241 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2b4f45d-b484-48d6-b6a6-43739a937815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-t7sjm\" (UID: \"a2b4f45d-b484-48d6-b6a6-43739a937815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm" Oct 30 13:14:11.529416 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:11.529294 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2b4f45d-b484-48d6-b6a6-43739a937815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-t7sjm\" (UID: \"a2b4f45d-b484-48d6-b6a6-43739a937815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm" Oct 30 13:14:11.529918 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:11.529891 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2b4f45d-b484-48d6-b6a6-43739a937815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-t7sjm\" (UID: \"a2b4f45d-b484-48d6-b6a6-43739a937815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm" Oct 30 13:14:11.531809 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:11.531779 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2b4f45d-b484-48d6-b6a6-43739a937815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-t7sjm\" (UID: \"a2b4f45d-b484-48d6-b6a6-43739a937815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm" Oct 30 13:14:11.557520 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:11.557484 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tzl7\" (UniqueName: \"kubernetes.io/projected/a2b4f45d-b484-48d6-b6a6-43739a937815-kube-api-access-9tzl7\") pod \"maas-api-revoke-sub-85237e45-t7sjm\" (UID: \"a2b4f45d-b484-48d6-b6a6-43739a937815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm" Oct 30 13:14:11.646068 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:11.646026 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm" Oct 30 13:14:11.954367 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:11.954340 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm"] Oct 30 13:14:11.957003 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:14:11.956973 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b4f45d_b484_48d6_b6a6_43739a937815.slice/crio-b9ebe0610a2df334980ef5a063b629de1047472996a79cdeaa8fed7c2bb92875 WatchSource:0}: Error finding container b9ebe0610a2df334980ef5a063b629de1047472996a79cdeaa8fed7c2bb92875: Status 404 returned error can't find the container with id b9ebe0610a2df334980ef5a063b629de1047472996a79cdeaa8fed7c2bb92875 Oct 30 13:14:12.551190 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:12.551109 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm" event={"ID":"a2b4f45d-b484-48d6-b6a6-43739a937815","Type":"ContainerStarted","Data":"cb0580a66bfe10ef2aa1d3e7ee968047c5979e562a4d462213aee023fad47b9a"} Oct 30 13:14:12.551190 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:12.551164 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm" event={"ID":"a2b4f45d-b484-48d6-b6a6-43739a937815","Type":"ContainerStarted","Data":"b9ebe0610a2df334980ef5a063b629de1047472996a79cdeaa8fed7c2bb92875"} Oct 30 13:14:12.595093 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:12.595031 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm" podStartSLOduration=1.594927486 podStartE2EDuration="1.594927486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:12.594079601 +0000 UTC m=+1346.417307878" watchObservedRunningTime="2026-10-30 13:14:12.594927486 +0000 UTC m=+1346.418155773" Oct 30 13:14:13.557704 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:13.557665 4503 generic.go:356] "Generic (PLEG): container finished" podID="a2b4f45d-b484-48d6-b6a6-43739a937815" containerID="cb0580a66bfe10ef2aa1d3e7ee968047c5979e562a4d462213aee023fad47b9a" exitCode=0 Oct 30 13:14:13.558146 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:13.557748 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm" event={"ID":"a2b4f45d-b484-48d6-b6a6-43739a937815","Type":"ContainerDied","Data":"cb0580a66bfe10ef2aa1d3e7ee968047c5979e562a4d462213aee023fad47b9a"} Oct 30 13:14:14.694571 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:14.694543 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm" Oct 30 13:14:14.864001 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:14.863901 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2b4f45d-b484-48d6-b6a6-43739a937815-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2b4f45d-b484-48d6-b6a6-43739a937815-maas-api-openshift-ca\") pod \"a2b4f45d-b484-48d6-b6a6-43739a937815\" (UID: \"a2b4f45d-b484-48d6-b6a6-43739a937815\") " Oct 30 13:14:14.864158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:14.864032 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2b4f45d-b484-48d6-b6a6-43739a937815-kube-api-access-9tzl7\" (UniqueName: \"kubernetes.io/projected/a2b4f45d-b484-48d6-b6a6-43739a937815-kube-api-access-9tzl7\") pod \"a2b4f45d-b484-48d6-b6a6-43739a937815\" (UID: \"a2b4f45d-b484-48d6-b6a6-43739a937815\") " Oct 30 13:14:14.864198 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:14.864151 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2b4f45d-b484-48d6-b6a6-43739a937815-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2b4f45d-b484-48d6-b6a6-43739a937815-maas-api-xks-ca\") pod \"a2b4f45d-b484-48d6-b6a6-43739a937815\" (UID: \"a2b4f45d-b484-48d6-b6a6-43739a937815\") " Oct 30 13:14:14.864326 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:14.864300 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b4f45d-b484-48d6-b6a6-43739a937815-maas-api-openshift-ca" pod "a2b4f45d-b484-48d6-b6a6-43739a937815" (UID: "a2b4f45d-b484-48d6-b6a6-43739a937815"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:14:14.864439 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:14.864424 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2b4f45d-b484-48d6-b6a6-43739a937815-maas-api-openshift-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:14:14.866176 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:14.866146 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2b4f45d-b484-48d6-b6a6-43739a937815-maas-api-xks-ca" pod "a2b4f45d-b484-48d6-b6a6-43739a937815" (UID: "a2b4f45d-b484-48d6-b6a6-43739a937815"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:14:14.866287 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:14.866203 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b4f45d-b484-48d6-b6a6-43739a937815-kube-api-access-9tzl7" pod "a2b4f45d-b484-48d6-b6a6-43739a937815" (UID: "a2b4f45d-b484-48d6-b6a6-43739a937815"). InnerVolumeSpecName "kube-api-access-9tzl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:14:14.965677 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:14.965649 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2b4f45d-b484-48d6-b6a6-43739a937815-maas-api-xks-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:14:14.965677 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:14.965675 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tzl7\" (UniqueName: \"kubernetes.io/projected/a2b4f45d-b484-48d6-b6a6-43739a937815-kube-api-access-9tzl7\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:14:15.569087 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:15.569058 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm" Oct 30 13:14:15.569087 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:15.569066 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm" event={"ID":"a2b4f45d-b484-48d6-b6a6-43739a937815","Type":"ContainerDied","Data":"b9ebe0610a2df334980ef5a063b629de1047472996a79cdeaa8fed7c2bb92875"} Oct 30 13:14:15.569308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:15.569109 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9ebe0610a2df334980ef5a063b629de1047472996a79cdeaa8fed7c2bb92875" Oct 30 13:14:23.060325 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.060285 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bd48f45d7-zp6j6"] Oct 30 13:14:23.060869 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.060512 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-bd48f45d7-zp6j6" podUID="8a39f600-f319-48ff-a1b1-37c523ccc255" containerName="manager" containerID="cri-o://6a753b9fe1b1432ae9193e095584668fc7d582efd7a171cbecf021912d83ec91" gracePeriod=30 Oct 30 13:14:23.320066 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.320004 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd48f45d7-zp6j6" Oct 30 13:14:23.451061 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.451029 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a39f600-f319-48ff-a1b1-37c523ccc255-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a39f600-f319-48ff-a1b1-37c523ccc255-metrics-certs\") pod \"8a39f600-f319-48ff-a1b1-37c523ccc255\" (UID: \"8a39f600-f319-48ff-a1b1-37c523ccc255\") " Oct 30 13:14:23.451252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.451109 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a39f600-f319-48ff-a1b1-37c523ccc255-kube-api-access-wv4d4\" (UniqueName: \"kubernetes.io/projected/8a39f600-f319-48ff-a1b1-37c523ccc255-kube-api-access-wv4d4\") pod \"8a39f600-f319-48ff-a1b1-37c523ccc255\" (UID: \"8a39f600-f319-48ff-a1b1-37c523ccc255\") " Oct 30 13:14:23.451252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.451152 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a39f600-f319-48ff-a1b1-37c523ccc255-webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a39f600-f319-48ff-a1b1-37c523ccc255-webhook-cert\") pod \"8a39f600-f319-48ff-a1b1-37c523ccc255\" (UID: \"8a39f600-f319-48ff-a1b1-37c523ccc255\") " Oct 30 13:14:23.453214 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.453179 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a39f600-f319-48ff-a1b1-37c523ccc255-kube-api-access-wv4d4" pod "8a39f600-f319-48ff-a1b1-37c523ccc255" (UID: "8a39f600-f319-48ff-a1b1-37c523ccc255"). InnerVolumeSpecName "kube-api-access-wv4d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:14:23.453214 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.453202 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a39f600-f319-48ff-a1b1-37c523ccc255-metrics-certs" pod "8a39f600-f319-48ff-a1b1-37c523ccc255" (UID: "8a39f600-f319-48ff-a1b1-37c523ccc255"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:14:23.453350 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.453224 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a39f600-f319-48ff-a1b1-37c523ccc255-webhook-cert" pod "8a39f600-f319-48ff-a1b1-37c523ccc255" (UID: "8a39f600-f319-48ff-a1b1-37c523ccc255"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:14:23.552675 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.552636 4503 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a39f600-f319-48ff-a1b1-37c523ccc255-webhook-cert\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:14:23.552675 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.552670 4503 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a39f600-f319-48ff-a1b1-37c523ccc255-metrics-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:14:23.552887 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.552688 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv4d4\" (UniqueName: \"kubernetes.io/projected/8a39f600-f319-48ff-a1b1-37c523ccc255-kube-api-access-wv4d4\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:14:23.619783 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.619691 4503 generic.go:356] "Generic (PLEG): container finished" podID="8a39f600-f319-48ff-a1b1-37c523ccc255" containerID="6a753b9fe1b1432ae9193e095584668fc7d582efd7a171cbecf021912d83ec91" exitCode=0 Oct 30 13:14:23.619783 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.619765 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd48f45d7-zp6j6" Oct 30 13:14:23.619985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.619780 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd48f45d7-zp6j6" event={"ID":"8a39f600-f319-48ff-a1b1-37c523ccc255","Type":"ContainerDied","Data":"6a753b9fe1b1432ae9193e095584668fc7d582efd7a171cbecf021912d83ec91"} Oct 30 13:14:23.619985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.619860 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd48f45d7-zp6j6" event={"ID":"8a39f600-f319-48ff-a1b1-37c523ccc255","Type":"ContainerDied","Data":"5bdd5dff217c8bca836c2bcb241e7305934fe3ee364b8197e9f27cc9939f8673"} Oct 30 13:14:23.619985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.619885 4503 scope.go:122] "RemoveContainer" containerID="6a753b9fe1b1432ae9193e095584668fc7d582efd7a171cbecf021912d83ec91" Oct 30 13:14:23.630298 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.630279 4503 scope.go:122] "RemoveContainer" containerID="6a753b9fe1b1432ae9193e095584668fc7d582efd7a171cbecf021912d83ec91" Oct 30 13:14:23.630547 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:14:23.630529 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a753b9fe1b1432ae9193e095584668fc7d582efd7a171cbecf021912d83ec91\": container with ID starting with 6a753b9fe1b1432ae9193e095584668fc7d582efd7a171cbecf021912d83ec91 not found: ID does not exist" containerID="6a753b9fe1b1432ae9193e095584668fc7d582efd7a171cbecf021912d83ec91" Oct 30 13:14:23.630596 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.630556 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a753b9fe1b1432ae9193e095584668fc7d582efd7a171cbecf021912d83ec91"} err="failed to get container status \"6a753b9fe1b1432ae9193e095584668fc7d582efd7a171cbecf021912d83ec91\": rpc error: code = NotFound desc = could not find container \"6a753b9fe1b1432ae9193e095584668fc7d582efd7a171cbecf021912d83ec91\": container with ID starting with 6a753b9fe1b1432ae9193e095584668fc7d582efd7a171cbecf021912d83ec91 not found: ID does not exist" Oct 30 13:14:23.681390 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.681354 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bd48f45d7-zp6j6"] Oct 30 13:14:23.693386 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:23.693358 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bd48f45d7-zp6j6"] Oct 30 13:14:24.360407 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:24.360378 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a39f600-f319-48ff-a1b1-37c523ccc255" path="/var/lib/kubelet/pods/8a39f600-f319-48ff-a1b1-37c523ccc255/volumes" Oct 30 13:14:24.943816 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:24.943781 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bd48f45d7-9tch9"] Oct 30 13:14:24.944344 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:24.944325 4503 cpu_manager.go:414] "containerMap: removing container" podUID="8a39f600-f319-48ff-a1b1-37c523ccc255" containerName="manager" Oct 30 13:14:24.944416 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:24.944347 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a39f600-f319-48ff-a1b1-37c523ccc255" containerName="manager" Oct 30 13:14:24.944416 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:24.944363 4503 cpu_manager.go:414] "containerMap: removing container" podUID="a2b4f45d-b484-48d6-b6a6-43739a937815" containerName="revoke-keys" Oct 30 13:14:24.944416 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:24.944368 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2b4f45d-b484-48d6-b6a6-43739a937815" containerName="revoke-keys" Oct 30 13:14:24.944516 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:24.944455 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a39f600-f319-48ff-a1b1-37c523ccc255" containerName="manager" Oct 30 13:14:24.944516 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:24.944464 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2b4f45d-b484-48d6-b6a6-43739a937815" containerName="revoke-keys" Oct 30 13:14:24.949079 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:24.949062 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd48f45d7-9tch9" Oct 30 13:14:24.961540 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:24.961520 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg5b6\" (UniqueName: \"kubernetes.io/projected/c1cd9a98-2a23-4361-a7c1-a12a456f8a69-kube-api-access-lg5b6\") pod \"maas-controller-bd48f45d7-9tch9\" (UID: \"c1cd9a98-2a23-4361-a7c1-a12a456f8a69\") " pod="opendatahub/maas-controller-bd48f45d7-9tch9" Oct 30 13:14:24.961649 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:24.961547 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1cd9a98-2a23-4361-a7c1-a12a456f8a69-metrics-certs\") pod \"maas-controller-bd48f45d7-9tch9\" (UID: \"c1cd9a98-2a23-4361-a7c1-a12a456f8a69\") " pod="opendatahub/maas-controller-bd48f45d7-9tch9" Oct 30 13:14:24.961698 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:24.961680 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1cd9a98-2a23-4361-a7c1-a12a456f8a69-webhook-cert\") pod \"maas-controller-bd48f45d7-9tch9\" (UID: \"c1cd9a98-2a23-4361-a7c1-a12a456f8a69\") " pod="opendatahub/maas-controller-bd48f45d7-9tch9" Oct 30 13:14:24.965654 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:24.965632 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 13:14:24.976622 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:24.976599 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 13:14:24.976727 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:24.976648 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-28684\"" Oct 30 13:14:24.976727 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:24.976662 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 13:14:24.980015 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:24.979994 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bd48f45d7-9tch9"] Oct 30 13:14:25.062335 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:25.062302 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1cd9a98-2a23-4361-a7c1-a12a456f8a69-webhook-cert\") pod \"maas-controller-bd48f45d7-9tch9\" (UID: \"c1cd9a98-2a23-4361-a7c1-a12a456f8a69\") " pod="opendatahub/maas-controller-bd48f45d7-9tch9" Oct 30 13:14:25.062480 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:25.062373 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg5b6\" (UniqueName: \"kubernetes.io/projected/c1cd9a98-2a23-4361-a7c1-a12a456f8a69-kube-api-access-lg5b6\") pod \"maas-controller-bd48f45d7-9tch9\" (UID: \"c1cd9a98-2a23-4361-a7c1-a12a456f8a69\") " pod="opendatahub/maas-controller-bd48f45d7-9tch9" Oct 30 13:14:25.062480 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:25.062398 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1cd9a98-2a23-4361-a7c1-a12a456f8a69-metrics-certs\") pod \"maas-controller-bd48f45d7-9tch9\" (UID: \"c1cd9a98-2a23-4361-a7c1-a12a456f8a69\") " pod="opendatahub/maas-controller-bd48f45d7-9tch9" Oct 30 13:14:25.064923 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:25.064898 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1cd9a98-2a23-4361-a7c1-a12a456f8a69-webhook-cert\") pod \"maas-controller-bd48f45d7-9tch9\" (UID: \"c1cd9a98-2a23-4361-a7c1-a12a456f8a69\") " pod="opendatahub/maas-controller-bd48f45d7-9tch9" Oct 30 13:14:25.065046 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:25.064969 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1cd9a98-2a23-4361-a7c1-a12a456f8a69-metrics-certs\") pod \"maas-controller-bd48f45d7-9tch9\" (UID: \"c1cd9a98-2a23-4361-a7c1-a12a456f8a69\") " pod="opendatahub/maas-controller-bd48f45d7-9tch9" Oct 30 13:14:25.087578 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:25.087550 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg5b6\" (UniqueName: \"kubernetes.io/projected/c1cd9a98-2a23-4361-a7c1-a12a456f8a69-kube-api-access-lg5b6\") pod \"maas-controller-bd48f45d7-9tch9\" (UID: \"c1cd9a98-2a23-4361-a7c1-a12a456f8a69\") " pod="opendatahub/maas-controller-bd48f45d7-9tch9" Oct 30 13:14:25.261761 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:25.261676 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd48f45d7-9tch9" Oct 30 13:14:25.571149 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:25.571122 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bd48f45d7-9tch9"] Oct 30 13:14:25.573028 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:14:25.572999 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1cd9a98_2a23_4361_a7c1_a12a456f8a69.slice/crio-7667414ba6d6dbd2eae649b928a42d24bd11b366e6eb364985f477fd5f0db6fa WatchSource:0}: Error finding container 7667414ba6d6dbd2eae649b928a42d24bd11b366e6eb364985f477fd5f0db6fa: Status 404 returned error can't find the container with id 7667414ba6d6dbd2eae649b928a42d24bd11b366e6eb364985f477fd5f0db6fa Oct 30 13:14:25.574399 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:25.574377 4503 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:14:25.632865 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:25.632823 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd48f45d7-9tch9" event={"ID":"c1cd9a98-2a23-4361-a7c1-a12a456f8a69","Type":"ContainerStarted","Data":"7667414ba6d6dbd2eae649b928a42d24bd11b366e6eb364985f477fd5f0db6fa"} Oct 30 13:14:26.138555 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.138530 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc"] Oct 30 13:14:26.143229 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.143212 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.151074 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.151054 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m"] Oct 30 13:14:26.154978 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.154947 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.155772 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.155751 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt"] Oct 30 13:14:26.159437 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.159422 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.166319 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.166274 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-g8svl\"" Oct 30 13:14:26.172268 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.172239 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc"] Oct 30 13:14:26.172899 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.172867 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07364b75-84b5-43af-8e63-c447d9f49f3c-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.173044 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.172913 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07364b75-84b5-43af-8e63-c447d9f49f3c-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.173044 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.172983 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkqwm\" (UniqueName: \"kubernetes.io/projected/07364b75-84b5-43af-8e63-c447d9f49f3c-kube-api-access-wkqwm\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.173150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173068 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d5e227-0a76-4cb2-b460-8409c61a9f12-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.173150 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173110 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d5e227-0a76-4cb2-b460-8409c61a9f12-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.173259 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173150 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07364b75-84b5-43af-8e63-c447d9f49f3c-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.173259 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173177 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6fc3d60-979c-4a16-9834-7c1083464b2c-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.173259 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173200 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d5e227-0a76-4cb2-b460-8409c61a9f12-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.173259 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173219 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fc3d60-979c-4a16-9834-7c1083464b2c-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.173259 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173235 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6fc3d60-979c-4a16-9834-7c1083464b2c-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.173259 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173252 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6fc3d60-979c-4a16-9834-7c1083464b2c-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.173851 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173278 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbvft\" (UniqueName: \"kubernetes.io/projected/f6fc3d60-979c-4a16-9834-7c1083464b2c-kube-api-access-vbvft\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.173851 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173301 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07364b75-84b5-43af-8e63-c447d9f49f3c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.173851 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173331 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07364b75-84b5-43af-8e63-c447d9f49f3c-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.173851 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173385 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d5e227-0a76-4cb2-b460-8409c61a9f12-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.173851 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173413 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07364b75-84b5-43af-8e63-c447d9f49f3c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.173851 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173456 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6fc3d60-979c-4a16-9834-7c1083464b2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.173851 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173503 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d5e227-0a76-4cb2-b460-8409c61a9f12-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.173851 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173538 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fc3d60-979c-4a16-9834-7c1083464b2c-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.173851 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173574 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6fc3d60-979c-4a16-9834-7c1083464b2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.173851 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173600 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07364b75-84b5-43af-8e63-c447d9f49f3c-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.173851 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173628 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07364b75-84b5-43af-8e63-c447d9f49f3c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.173851 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173656 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6fc3d60-979c-4a16-9834-7c1083464b2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.173851 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173703 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d5e227-0a76-4cb2-b460-8409c61a9f12-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.173851 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173731 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d5e227-0a76-4cb2-b460-8409c61a9f12-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.173851 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173760 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d5e227-0a76-4cb2-b460-8409c61a9f12-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.173851 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.173792 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2lm5\" (UniqueName: \"kubernetes.io/projected/d4d5e227-0a76-4cb2-b460-8409c61a9f12-kube-api-access-x2lm5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.177186 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.177155 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r"] Oct 30 13:14:26.181236 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.181214 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.186737 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.186714 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m"] Oct 30 13:14:26.189101 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.189082 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt"] Oct 30 13:14:26.207303 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.207276 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r"] Oct 30 13:14:26.275044 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275011 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.275044 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275043 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.275305 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275074 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d5e227-0a76-4cb2-b460-8409c61a9f12-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.275305 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275143 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d5e227-0a76-4cb2-b460-8409c61a9f12-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.275305 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275180 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07364b75-84b5-43af-8e63-c447d9f49f3c-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.275305 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275206 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6fc3d60-979c-4a16-9834-7c1083464b2c-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.275305 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275231 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.275305 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275262 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d5e227-0a76-4cb2-b460-8409c61a9f12-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.275305 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275289 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fc3d60-979c-4a16-9834-7c1083464b2c-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.275700 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275412 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6fc3d60-979c-4a16-9834-7c1083464b2c-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.275700 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275455 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6fc3d60-979c-4a16-9834-7c1083464b2c-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.275700 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275482 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbvft\" (UniqueName: \"kubernetes.io/projected/f6fc3d60-979c-4a16-9834-7c1083464b2c-kube-api-access-vbvft\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.275700 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275510 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07364b75-84b5-43af-8e63-c447d9f49f3c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.275700 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275529 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d5e227-0a76-4cb2-b460-8409c61a9f12-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.275700 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275548 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.275700 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275613 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07364b75-84b5-43af-8e63-c447d9f49f3c-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.275700 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275621 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d5e227-0a76-4cb2-b460-8409c61a9f12-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.275700 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275658 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d5e227-0a76-4cb2-b460-8409c61a9f12-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.275700 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275684 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fc3d60-979c-4a16-9834-7c1083464b2c-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275690 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07364b75-84b5-43af-8e63-c447d9f49f3c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275743 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6fc3d60-979c-4a16-9834-7c1083464b2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275773 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275801 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d5e227-0a76-4cb2-b460-8409c61a9f12-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275833 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fc3d60-979c-4a16-9834-7c1083464b2c-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275854 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6fc3d60-979c-4a16-9834-7c1083464b2c-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275868 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6fc3d60-979c-4a16-9834-7c1083464b2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275872 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07364b75-84b5-43af-8e63-c447d9f49f3c-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275906 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07364b75-84b5-43af-8e63-c447d9f49f3c-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275934 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d5e227-0a76-4cb2-b460-8409c61a9f12-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275977 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07364b75-84b5-43af-8e63-c447d9f49f3c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.276009 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6fc3d60-979c-4a16-9834-7c1083464b2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.276043 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.276073 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9m9c\" (UniqueName: \"kubernetes.io/projected/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-kube-api-access-n9m9c\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.276111 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d5e227-0a76-4cb2-b460-8409c61a9f12-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.276141 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d5e227-0a76-4cb2-b460-8409c61a9f12-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.276166 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d5e227-0a76-4cb2-b460-8409c61a9f12-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.276200 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2lm5\" (UniqueName: \"kubernetes.io/projected/d4d5e227-0a76-4cb2-b460-8409c61a9f12-kube-api-access-x2lm5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.276207 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07364b75-84b5-43af-8e63-c447d9f49f3c-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.276252 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.275661 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07364b75-84b5-43af-8e63-c447d9f49f3c-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.277390 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.276628 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07364b75-84b5-43af-8e63-c447d9f49f3c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.277390 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.276636 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6fc3d60-979c-4a16-9834-7c1083464b2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.277390 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.276882 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6fc3d60-979c-4a16-9834-7c1083464b2c-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.277390 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.277162 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d5e227-0a76-4cb2-b460-8409c61a9f12-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.277390 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.277174 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fc3d60-979c-4a16-9834-7c1083464b2c-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.277390 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.277317 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d5e227-0a76-4cb2-b460-8409c61a9f12-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.277390 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.277317 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07364b75-84b5-43af-8e63-c447d9f49f3c-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.277390 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.277377 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07364b75-84b5-43af-8e63-c447d9f49f3c-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.277820 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.277418 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkqwm\" (UniqueName: \"kubernetes.io/projected/07364b75-84b5-43af-8e63-c447d9f49f3c-kube-api-access-wkqwm\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.277820 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.277451 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.277820 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.277479 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.277820 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.277800 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07364b75-84b5-43af-8e63-c447d9f49f3c-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.278042 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.277912 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d5e227-0a76-4cb2-b460-8409c61a9f12-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.278337 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.278317 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07364b75-84b5-43af-8e63-c447d9f49f3c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.278669 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.278650 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6fc3d60-979c-4a16-9834-7c1083464b2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.279253 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.279229 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d5e227-0a76-4cb2-b460-8409c61a9f12-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.279530 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.279509 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07364b75-84b5-43af-8e63-c447d9f49f3c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.280227 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.280203 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6fc3d60-979c-4a16-9834-7c1083464b2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.302576 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.302550 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbvft\" (UniqueName: \"kubernetes.io/projected/f6fc3d60-979c-4a16-9834-7c1083464b2c-kube-api-access-vbvft\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.303807 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.303780 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkqwm\" (UniqueName: \"kubernetes.io/projected/07364b75-84b5-43af-8e63-c447d9f49f3c-kube-api-access-wkqwm\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.303921 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.303817 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d5e227-0a76-4cb2-b460-8409c61a9f12-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.304018 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.303993 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07364b75-84b5-43af-8e63-c447d9f49f3c-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5dtc\" (UID: \"07364b75-84b5-43af-8e63-c447d9f49f3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.304345 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.304319 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2lm5\" (UniqueName: \"kubernetes.io/projected/d4d5e227-0a76-4cb2-b460-8409c61a9f12-kube-api-access-x2lm5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8ndvt\" (UID: \"d4d5e227-0a76-4cb2-b460-8409c61a9f12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.304768 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.304751 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6fc3d60-979c-4a16-9834-7c1083464b2c-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gdg2m\" (UID: \"f6fc3d60-979c-4a16-9834-7c1083464b2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.378986 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.378865 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.378986 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.378910 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.379178 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.378983 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.379178 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.379016 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.379178 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.379150 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.379315 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.379211 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.379315 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.379268 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.379415 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.379336 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.379415 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.379369 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9m9c\" (UniqueName: \"kubernetes.io/projected/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-kube-api-access-n9m9c\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.379558 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.379528 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.379620 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.379560 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.379673 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.379633 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.379746 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.379727 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.379908 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.379888 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.381292 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.381273 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.381619 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.381597 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.403671 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.403644 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.403903 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.403883 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9m9c\" (UniqueName: \"kubernetes.io/projected/dfc85b9a-8b4e-44fb-8de4-635cf391a3cd-kube-api-access-n9m9c\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9h2r\" (UID: \"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.461465 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.461426 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:26.475392 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.475356 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:26.485813 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.485783 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:26.497045 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.497008 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:26.648906 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.648867 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd48f45d7-9tch9" event={"ID":"c1cd9a98-2a23-4361-a7c1-a12a456f8a69","Type":"ContainerStarted","Data":"639cb4c33243394cf3d872bc5ffcbf650418e87e7a43b90d8f5971f0bb9d3ab7"} Oct 30 13:14:26.649358 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.649073 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bd48f45d7-9tch9" Oct 30 13:14:26.700306 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.700252 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-bd48f45d7-9tch9" podStartSLOduration=2.184753821 podStartE2EDuration="2.700237843s" totalImagesPullingTime="515.484022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:26.698536172 +0000 UTC m=+1360.521764449" watchObservedRunningTime="2026-10-30 13:14:26.700237843 +0000 UTC m=+1360.523466120" Oct 30 13:14:26.800618 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.800583 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc"] Oct 30 13:14:26.802297 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:14:26.802269 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07364b75_84b5_43af_8e63_c447d9f49f3c.slice/crio-bda785f15faf01cf49bf24d7cfdb1512eee6c47a48ad423c1e00095a360f291c WatchSource:0}: Error finding container bda785f15faf01cf49bf24d7cfdb1512eee6c47a48ad423c1e00095a360f291c: Status 404 returned error can't find the container with id bda785f15faf01cf49bf24d7cfdb1512eee6c47a48ad423c1e00095a360f291c Oct 30 13:14:26.803932 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.803914 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m"] Oct 30 13:14:26.805543 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.805510 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:26.805665 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.805585 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:26.805665 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.805630 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:26.805862 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:14:26.805841 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6fc3d60_979c_4a16_9834_7c1083464b2c.slice/crio-caae71fd02f62242b0810d3e6fc4764dbdafb2ffc0b0e5fc6d9936da9f394727 WatchSource:0}: Error finding container caae71fd02f62242b0810d3e6fc4764dbdafb2ffc0b0e5fc6d9936da9f394727: Status 404 returned error can't find the container with id caae71fd02f62242b0810d3e6fc4764dbdafb2ffc0b0e5fc6d9936da9f394727 Oct 30 13:14:26.808031 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.808005 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:26.808086 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.808061 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:26.808130 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.808100 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:26.821361 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.821180 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt"] Oct 30 13:14:26.822188 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:14:26.822155 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d5e227_0a76_4cb2_b460_8409c61a9f12.slice/crio-89459185b19e1c4c43a17b1cf30b5019fcf6495bac6ed5c9c148cdd3105db418 WatchSource:0}: Error finding container 89459185b19e1c4c43a17b1cf30b5019fcf6495bac6ed5c9c148cdd3105db418: Status 404 returned error can't find the container with id 89459185b19e1c4c43a17b1cf30b5019fcf6495bac6ed5c9c148cdd3105db418 Oct 30 13:14:26.825096 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.825068 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:26.825159 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.825142 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:26.825208 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.825192 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:26.827022 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.826996 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r"] Oct 30 13:14:26.827453 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:14:26.827385 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc85b9a_8b4e_44fb_8de4_635cf391a3cd.slice/crio-e391fec010068ca21d0dfdd26d119505fe464f04f824861368f705447c1762a8 WatchSource:0}: Error finding container e391fec010068ca21d0dfdd26d119505fe464f04f824861368f705447c1762a8: Status 404 returned error can't find the container with id e391fec010068ca21d0dfdd26d119505fe464f04f824861368f705447c1762a8 Oct 30 13:14:26.833240 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.833215 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:26.833326 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.833267 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:26.833326 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:26.833295 4503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:27.714966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:27.702061 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" event={"ID":"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd","Type":"ContainerStarted","Data":"6146a2901bbb15a46244873cbc39e994a42c1c070ca08369a1d6870efe45250b"} Oct 30 13:14:27.714966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:27.702133 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" event={"ID":"dfc85b9a-8b4e-44fb-8de4-635cf391a3cd","Type":"ContainerStarted","Data":"e391fec010068ca21d0dfdd26d119505fe464f04f824861368f705447c1762a8"} Oct 30 13:14:27.727962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:27.724884 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" event={"ID":"07364b75-84b5-43af-8e63-c447d9f49f3c","Type":"ContainerStarted","Data":"948e75b16151753d331f356a0d4088c157d88607680a1d542404b34d8bfa8407"} Oct 30 13:14:27.727962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:27.724972 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" event={"ID":"07364b75-84b5-43af-8e63-c447d9f49f3c","Type":"ContainerStarted","Data":"bda785f15faf01cf49bf24d7cfdb1512eee6c47a48ad423c1e00095a360f291c"} Oct 30 13:14:27.739651 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:27.738856 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" event={"ID":"d4d5e227-0a76-4cb2-b460-8409c61a9f12","Type":"ContainerStarted","Data":"7d8196dd63d57146878d6be1b947d76bf58089775291fd18de20dacf16d1c630"} Oct 30 13:14:27.739651 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:27.738921 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" event={"ID":"d4d5e227-0a76-4cb2-b460-8409c61a9f12","Type":"ContainerStarted","Data":"89459185b19e1c4c43a17b1cf30b5019fcf6495bac6ed5c9c148cdd3105db418"} Oct 30 13:14:27.760963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:27.748313 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" event={"ID":"f6fc3d60-979c-4a16-9834-7c1083464b2c","Type":"ContainerStarted","Data":"195698a46d77455cc40ac2a65475b6a5a6858cf1000ceb657d12842140a29e74"} Oct 30 13:14:27.760963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:27.748368 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" event={"ID":"f6fc3d60-979c-4a16-9834-7c1083464b2c","Type":"ContainerStarted","Data":"caae71fd02f62242b0810d3e6fc4764dbdafb2ffc0b0e5fc6d9936da9f394727"} Oct 30 13:14:27.832963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:27.832005 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" podStartSLOduration=1.831984568 podStartE2EDuration="1.831984568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:27.806550886 +0000 UTC m=+1361.629779176" watchObservedRunningTime="2026-10-30 13:14:27.831984568 +0000 UTC m=+1361.655212846" Oct 30 13:14:27.887962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:27.887507 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" podStartSLOduration=1.887485951 podStartE2EDuration="1.887485951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:27.887326086 +0000 UTC m=+1361.710554363" watchObservedRunningTime="2026-10-30 13:14:27.887485951 +0000 UTC m=+1361.710714231" Oct 30 13:14:28.020954 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:28.020809 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" podStartSLOduration=2.020787774 podStartE2EDuration="2.020787774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:27.949861744 +0000 UTC m=+1361.773090022" watchObservedRunningTime="2026-10-30 13:14:28.020787774 +0000 UTC m=+1361.844016045" Oct 30 13:14:28.470961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:28.469538 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:28.483964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:28.482142 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:28.483964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:28.482327 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gdg2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 13:14:28.483964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:28.482375 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" podUID="f6fc3d60-979c-4a16-9834-7c1083464b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 13:14:28.483964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:28.482750 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k5dtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 13:14:28.483964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:28.482800 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" podUID="07364b75-84b5-43af-8e63-c447d9f49f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 13:14:28.488963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:28.488641 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:28.488963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:28.488765 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8ndvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 13:14:28.488963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:28.488806 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" podUID="d4d5e227-0a76-4cb2-b460-8409c61a9f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 13:14:28.513965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:28.509427 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:28.518556 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:28.518519 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x9h2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:14:28.518828 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:28.518800 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" podUID="dfc85b9a-8b4e-44fb-8de4-635cf391a3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 13:14:29.466964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:29.466151 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k5dtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 13:14:29.466964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:29.466214 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" podUID="07364b75-84b5-43af-8e63-c447d9f49f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 13:14:29.480968 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:29.478575 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gdg2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 13:14:29.480968 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:29.478642 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" podUID="f6fc3d60-979c-4a16-9834-7c1083464b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 13:14:29.488159 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:29.488129 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8ndvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 13:14:29.488411 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:29.488386 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" podUID="d4d5e227-0a76-4cb2-b460-8409c61a9f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 13:14:29.503411 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:29.503375 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x9h2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:14:29.503679 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:29.503650 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" podUID="dfc85b9a-8b4e-44fb-8de4-635cf391a3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 13:14:29.929965 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:14:29.928244 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:14:30.465964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:30.465371 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k5dtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 13:14:30.465964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:30.465435 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" podUID="07364b75-84b5-43af-8e63-c447d9f49f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 13:14:30.480963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:30.478328 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gdg2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 13:14:30.480963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:30.478391 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" podUID="f6fc3d60-979c-4a16-9834-7c1083464b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 13:14:30.492974 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:30.489854 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8ndvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 13:14:30.492974 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:30.489911 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" podUID="d4d5e227-0a76-4cb2-b460-8409c61a9f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 13:14:30.502964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:30.499330 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x9h2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:14:30.502964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:30.499387 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" podUID="dfc85b9a-8b4e-44fb-8de4-635cf391a3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 13:14:31.464964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:31.463353 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k5dtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 13:14:31.464964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:31.463419 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" podUID="07364b75-84b5-43af-8e63-c447d9f49f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 13:14:31.489965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:31.489730 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gdg2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 13:14:31.489965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:31.489798 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" podUID="f6fc3d60-979c-4a16-9834-7c1083464b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 13:14:31.493966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:31.490635 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8ndvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 13:14:31.493966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:31.490682 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" podUID="d4d5e227-0a76-4cb2-b460-8409c61a9f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 13:14:31.501385 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:31.501343 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x9h2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:14:31.501661 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:31.501623 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" podUID="dfc85b9a-8b4e-44fb-8de4-635cf391a3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 13:14:32.465433 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:32.465394 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k5dtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 13:14:32.465729 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:32.465700 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" podUID="07364b75-84b5-43af-8e63-c447d9f49f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 13:14:32.480345 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:32.480306 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gdg2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 13:14:32.480596 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:32.480568 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" podUID="f6fc3d60-979c-4a16-9834-7c1083464b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 13:14:32.486345 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:32.486312 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8ndvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 13:14:32.486575 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:32.486549 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" podUID="d4d5e227-0a76-4cb2-b460-8409c61a9f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 13:14:32.501431 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:32.501385 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x9h2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:14:32.502073 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:32.502038 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" podUID="dfc85b9a-8b4e-44fb-8de4-635cf391a3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 13:14:33.462455 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:33.462412 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k5dtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 13:14:33.462754 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:33.462723 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" podUID="07364b75-84b5-43af-8e63-c447d9f49f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 13:14:33.476373 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:33.476332 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gdg2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 13:14:33.476636 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:33.476608 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" podUID="f6fc3d60-979c-4a16-9834-7c1083464b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 13:14:33.486427 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:33.486386 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8ndvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 13:14:33.486617 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:33.486451 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" podUID="d4d5e227-0a76-4cb2-b460-8409c61a9f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 13:14:33.497919 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:33.497883 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x9h2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:14:33.498102 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:33.497964 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" podUID="dfc85b9a-8b4e-44fb-8de4-635cf391a3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 13:14:34.462820 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:34.462775 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k5dtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 13:14:34.463379 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:34.462851 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" podUID="07364b75-84b5-43af-8e63-c447d9f49f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 13:14:34.476276 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:34.476229 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gdg2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 13:14:34.476465 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:34.476303 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" podUID="f6fc3d60-979c-4a16-9834-7c1083464b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 13:14:34.486148 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:34.486117 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8ndvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 13:14:34.486395 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:34.486357 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" podUID="d4d5e227-0a76-4cb2-b460-8409c61a9f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 13:14:34.497474 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:34.497443 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x9h2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:14:34.497634 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:34.497500 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" podUID="dfc85b9a-8b4e-44fb-8de4-635cf391a3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 13:14:36.293355 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:36.291811 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:36.293355 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:36.293186 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:36.294199 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:36.294152 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" Oct 30 13:14:36.332717 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:36.332681 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:36.333146 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:36.333064 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:36.335671 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:36.335647 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" Oct 30 13:14:36.356258 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:36.356196 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" podStartSLOduration=10.356177286 podStartE2EDuration="10.356177286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:28.022037404 +0000 UTC m=+1361.845265678" watchObservedRunningTime="2026-10-30 13:14:36.356177286 +0000 UTC m=+1370.179405565" Oct 30 13:14:36.375462 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:36.375433 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:36.376437 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:36.376323 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:36.377160 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:36.377134 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" Oct 30 13:14:36.531556 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:36.531503 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x9h2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:14:36.531841 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:36.531805 4503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" podUID="dfc85b9a-8b4e-44fb-8de4-635cf391a3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:14:36.586606 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:36.586509 4503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:36.587067 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:36.587026 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:36.587657 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:36.587635 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" Oct 30 13:14:37.750220 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:14:37.750184 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bd48f45d7-9tch9" Oct 30 13:14:59.702185 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:14:59.702096 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:15:00.164778 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:00.164737 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x"] Oct 30 13:15:00.170072 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:00.170044 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" Oct 30 13:15:00.190003 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:00.189968 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x"] Oct 30 13:15:00.197982 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:00.197917 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:15:00.307492 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:00.307457 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d87698-da9f-4805-9037-79a9c4c7b6b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889435-mkz4x\" (UID: \"12d87698-da9f-4805-9037-79a9c4c7b6b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" Oct 30 13:15:00.307669 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:00.307526 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgjrm\" (UniqueName: \"kubernetes.io/projected/12d87698-da9f-4805-9037-79a9c4c7b6b6-kube-api-access-bgjrm\") pod \"maas-api-key-cleanup-29889435-mkz4x\" (UID: \"12d87698-da9f-4805-9037-79a9c4c7b6b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" Oct 30 13:15:00.408947 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:00.408898 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d87698-da9f-4805-9037-79a9c4c7b6b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889435-mkz4x\" (UID: \"12d87698-da9f-4805-9037-79a9c4c7b6b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" Oct 30 13:15:00.409135 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:00.409002 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgjrm\" (UniqueName: \"kubernetes.io/projected/12d87698-da9f-4805-9037-79a9c4c7b6b6-kube-api-access-bgjrm\") pod \"maas-api-key-cleanup-29889435-mkz4x\" (UID: \"12d87698-da9f-4805-9037-79a9c4c7b6b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" Oct 30 13:15:00.409552 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:00.409526 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d87698-da9f-4805-9037-79a9c4c7b6b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889435-mkz4x\" (UID: \"12d87698-da9f-4805-9037-79a9c4c7b6b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" Oct 30 13:15:00.435673 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:00.435611 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgjrm\" (UniqueName: \"kubernetes.io/projected/12d87698-da9f-4805-9037-79a9c4c7b6b6-kube-api-access-bgjrm\") pod \"maas-api-key-cleanup-29889435-mkz4x\" (UID: \"12d87698-da9f-4805-9037-79a9c4c7b6b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" Oct 30 13:15:00.485692 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:00.485658 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" Oct 30 13:15:00.786625 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:00.786587 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x"] Oct 30 13:15:00.788295 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:15:00.788265 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d87698_da9f_4805_9037_79a9c4c7b6b6.slice/crio-a5088ef55ad3b989105bbccefa0b61c379eeefb4677d1a0912a891205daf5fa8 WatchSource:0}: Error finding container a5088ef55ad3b989105bbccefa0b61c379eeefb4677d1a0912a891205daf5fa8: Status 404 returned error can't find the container with id a5088ef55ad3b989105bbccefa0b61c379eeefb4677d1a0912a891205daf5fa8 Oct 30 13:15:01.079184 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:01.079148 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" event={"ID":"12d87698-da9f-4805-9037-79a9c4c7b6b6","Type":"ContainerStarted","Data":"9679b4eeb6e84961f06eda4de73fc5a2519c6b15ff10e0d369d5701c3704b5de"} Oct 30 13:15:01.079322 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:01.079209 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" event={"ID":"12d87698-da9f-4805-9037-79a9c4c7b6b6","Type":"ContainerStarted","Data":"a5088ef55ad3b989105bbccefa0b61c379eeefb4677d1a0912a891205daf5fa8"} Oct 30 13:15:01.134871 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:01.134820 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" podStartSLOduration=1.134802481 podStartE2EDuration="1.134802481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:01.13189733 +0000 UTC m=+1394.955125606" watchObservedRunningTime="2026-10-30 13:15:01.134802481 +0000 UTC m=+1394.958030759" Oct 30 13:15:02.086697 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:02.086664 4503 generic.go:356] "Generic (PLEG): container finished" podID="12d87698-da9f-4805-9037-79a9c4c7b6b6" containerID="9679b4eeb6e84961f06eda4de73fc5a2519c6b15ff10e0d369d5701c3704b5de" exitCode=60 Oct 30 13:15:02.087229 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:02.086720 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" event={"ID":"12d87698-da9f-4805-9037-79a9c4c7b6b6","Type":"ContainerDied","Data":"9679b4eeb6e84961f06eda4de73fc5a2519c6b15ff10e0d369d5701c3704b5de"} Oct 30 13:15:02.087229 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:02.087084 4503 scope.go:122] "RemoveContainer" containerID="9679b4eeb6e84961f06eda4de73fc5a2519c6b15ff10e0d369d5701c3704b5de" Oct 30 13:15:03.094434 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:03.094401 4503 generic.go:356] "Generic (PLEG): container finished" podID="12d87698-da9f-4805-9037-79a9c4c7b6b6" containerID="a2b2467222a7d29d43e4d5e405cbfddf8e16be31a748d7152e2d6ef66cbdb50e" exitCode=60 Oct 30 13:15:03.094966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:03.094492 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" event={"ID":"12d87698-da9f-4805-9037-79a9c4c7b6b6","Type":"ContainerDied","Data":"a2b2467222a7d29d43e4d5e405cbfddf8e16be31a748d7152e2d6ef66cbdb50e"} Oct 30 13:15:03.094966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:03.094553 4503 scope.go:122] "RemoveContainer" containerID="9679b4eeb6e84961f06eda4de73fc5a2519c6b15ff10e0d369d5701c3704b5de" Oct 30 13:15:03.094966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:03.094805 4503 scope.go:122] "RemoveContainer" containerID="a2b2467222a7d29d43e4d5e405cbfddf8e16be31a748d7152e2d6ef66cbdb50e" Oct 30 13:15:03.095124 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:15:03.095091 4503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889435-mkz4x_odh-ai-gateway-infra(12d87698-da9f-4805-9037-79a9c4c7b6b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" podUID="12d87698-da9f-4805-9037-79a9c4c7b6b6" Oct 30 13:15:04.103909 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:04.103881 4503 scope.go:122] "RemoveContainer" containerID="a2b2467222a7d29d43e4d5e405cbfddf8e16be31a748d7152e2d6ef66cbdb50e" Oct 30 13:15:04.104315 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:15:04.104144 4503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889435-mkz4x_odh-ai-gateway-infra(12d87698-da9f-4805-9037-79a9c4c7b6b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" podUID="12d87698-da9f-4805-9037-79a9c4c7b6b6" Oct 30 13:15:17.354525 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:17.354494 4503 scope.go:122] "RemoveContainer" containerID="a2b2467222a7d29d43e4d5e405cbfddf8e16be31a748d7152e2d6ef66cbdb50e" Oct 30 13:15:18.305696 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:18.305661 4503 generic.go:356] "Generic (PLEG): container finished" podID="12d87698-da9f-4805-9037-79a9c4c7b6b6" containerID="acc11bfe6c9da933a6bf7bccbfc288890bac45871e82f6318a078432c122fa01" exitCode=60 Oct 30 13:15:18.305860 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:18.305731 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" event={"ID":"12d87698-da9f-4805-9037-79a9c4c7b6b6","Type":"ContainerDied","Data":"acc11bfe6c9da933a6bf7bccbfc288890bac45871e82f6318a078432c122fa01"} Oct 30 13:15:18.305860 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:18.305790 4503 scope.go:122] "RemoveContainer" containerID="a2b2467222a7d29d43e4d5e405cbfddf8e16be31a748d7152e2d6ef66cbdb50e" Oct 30 13:15:18.306238 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:18.306223 4503 scope.go:122] "RemoveContainer" containerID="acc11bfe6c9da933a6bf7bccbfc288890bac45871e82f6318a078432c122fa01" Oct 30 13:15:18.306490 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:15:18.306472 4503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889435-mkz4x_odh-ai-gateway-infra(12d87698-da9f-4805-9037-79a9c4c7b6b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" podUID="12d87698-da9f-4805-9037-79a9c4c7b6b6" Oct 30 13:15:19.367831 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:19.367773 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x"] Oct 30 13:15:19.878702 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:19.878402 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" Oct 30 13:15:19.988979 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:19.982390 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d87698-da9f-4805-9037-79a9c4c7b6b6-kube-api-access-bgjrm\" (UniqueName: \"kubernetes.io/projected/12d87698-da9f-4805-9037-79a9c4c7b6b6-kube-api-access-bgjrm\") pod \"12d87698-da9f-4805-9037-79a9c4c7b6b6\" (UID: \"12d87698-da9f-4805-9037-79a9c4c7b6b6\") " Oct 30 13:15:19.988979 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:19.982449 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12d87698-da9f-4805-9037-79a9c4c7b6b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d87698-da9f-4805-9037-79a9c4c7b6b6-maas-api-service-ca\") pod \"12d87698-da9f-4805-9037-79a9c4c7b6b6\" (UID: \"12d87698-da9f-4805-9037-79a9c4c7b6b6\") " Oct 30 13:15:19.989803 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:19.989452 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d87698-da9f-4805-9037-79a9c4c7b6b6-maas-api-service-ca" pod "12d87698-da9f-4805-9037-79a9c4c7b6b6" (UID: "12d87698-da9f-4805-9037-79a9c4c7b6b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:20.038980 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:20.031488 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d87698-da9f-4805-9037-79a9c4c7b6b6-kube-api-access-bgjrm" pod "12d87698-da9f-4805-9037-79a9c4c7b6b6" (UID: "12d87698-da9f-4805-9037-79a9c4c7b6b6"). InnerVolumeSpecName "kube-api-access-bgjrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:20.086677 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:20.086584 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgjrm\" (UniqueName: \"kubernetes.io/projected/12d87698-da9f-4805-9037-79a9c4c7b6b6-kube-api-access-bgjrm\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:15:20.086677 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:20.086625 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d87698-da9f-4805-9037-79a9c4c7b6b6-maas-api-service-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:15:20.457961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:20.457197 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" Oct 30 13:15:20.502293 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:20.500876 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x" event={"ID":"12d87698-da9f-4805-9037-79a9c4c7b6b6","Type":"ContainerDied","Data":"a5088ef55ad3b989105bbccefa0b61c379eeefb4677d1a0912a891205daf5fa8"} Oct 30 13:15:20.502293 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:20.500987 4503 scope.go:122] "RemoveContainer" containerID="acc11bfe6c9da933a6bf7bccbfc288890bac45871e82f6318a078432c122fa01" Oct 30 13:15:20.718026 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:20.704071 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x"] Oct 30 13:15:20.741976 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:20.741825 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-mkz4x"] Oct 30 13:15:22.361777 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:22.361740 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12d87698-da9f-4805-9037-79a9c4c7b6b6" path="/var/lib/kubelet/pods/12d87698-da9f-4805-9037-79a9c4c7b6b6/volumes" Oct 30 13:15:24.931122 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:24.931089 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc"] Oct 30 13:15:24.931515 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:24.931503 4503 cpu_manager.go:414] "containerMap: removing container" podUID="12d87698-da9f-4805-9037-79a9c4c7b6b6" containerName="cleanup" Oct 30 13:15:24.931567 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:24.931516 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d87698-da9f-4805-9037-79a9c4c7b6b6" containerName="cleanup" Oct 30 13:15:24.931567 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:24.931545 4503 cpu_manager.go:414] "containerMap: removing container" podUID="12d87698-da9f-4805-9037-79a9c4c7b6b6" containerName="cleanup" Oct 30 13:15:24.931567 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:24.931551 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d87698-da9f-4805-9037-79a9c4c7b6b6" containerName="cleanup" Oct 30 13:15:24.931567 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:24.931560 4503 cpu_manager.go:414] "containerMap: removing container" podUID="12d87698-da9f-4805-9037-79a9c4c7b6b6" containerName="cleanup" Oct 30 13:15:24.931567 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:24.931565 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d87698-da9f-4805-9037-79a9c4c7b6b6" containerName="cleanup" Oct 30 13:15:24.931724 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:24.931644 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d87698-da9f-4805-9037-79a9c4c7b6b6" containerName="cleanup" Oct 30 13:15:24.931724 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:24.931656 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d87698-da9f-4805-9037-79a9c4c7b6b6" containerName="cleanup" Oct 30 13:15:24.934663 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:24.934647 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc" Oct 30 13:15:24.961066 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:24.961034 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc"] Oct 30 13:15:24.968953 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:24.968905 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:15:24.969119 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:24.968966 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:15:25.057000 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:25.056962 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c015946-f285-4a73-ab87-d747da2b3509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2z7cc\" (UID: \"6c015946-f285-4a73-ab87-d747da2b3509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc" Oct 30 13:15:25.057221 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:25.057102 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhn6w\" (UniqueName: \"kubernetes.io/projected/6c015946-f285-4a73-ab87-d747da2b3509-kube-api-access-xhn6w\") pod \"maas-api-revoke-sub-2e1f5b5f-2z7cc\" (UID: \"6c015946-f285-4a73-ab87-d747da2b3509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc" Oct 30 13:15:25.057221 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:25.057135 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c015946-f285-4a73-ab87-d747da2b3509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2z7cc\" (UID: \"6c015946-f285-4a73-ab87-d747da2b3509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc" Oct 30 13:15:25.158354 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:25.158321 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c015946-f285-4a73-ab87-d747da2b3509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2z7cc\" (UID: \"6c015946-f285-4a73-ab87-d747da2b3509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc" Oct 30 13:15:25.158550 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:25.158397 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhn6w\" (UniqueName: \"kubernetes.io/projected/6c015946-f285-4a73-ab87-d747da2b3509-kube-api-access-xhn6w\") pod \"maas-api-revoke-sub-2e1f5b5f-2z7cc\" (UID: \"6c015946-f285-4a73-ab87-d747da2b3509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc" Oct 30 13:15:25.158550 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:25.158433 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c015946-f285-4a73-ab87-d747da2b3509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2z7cc\" (UID: \"6c015946-f285-4a73-ab87-d747da2b3509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc" Oct 30 13:15:25.159093 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:25.159073 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c015946-f285-4a73-ab87-d747da2b3509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2z7cc\" (UID: \"6c015946-f285-4a73-ab87-d747da2b3509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc" Oct 30 13:15:25.160961 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:25.160915 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c015946-f285-4a73-ab87-d747da2b3509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2z7cc\" (UID: \"6c015946-f285-4a73-ab87-d747da2b3509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc" Oct 30 13:15:25.183423 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:25.183354 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhn6w\" (UniqueName: \"kubernetes.io/projected/6c015946-f285-4a73-ab87-d747da2b3509-kube-api-access-xhn6w\") pod \"maas-api-revoke-sub-2e1f5b5f-2z7cc\" (UID: \"6c015946-f285-4a73-ab87-d747da2b3509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc" Oct 30 13:15:25.246261 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:25.246228 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc" Oct 30 13:15:25.552173 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:25.552145 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc"] Oct 30 13:15:25.554316 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:15:25.554283 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c015946_f285_4a73_ab87_d747da2b3509.slice/crio-2ad74d681897735446021251b3195a2c03071d8cd800023045ce4e769e67d3d5 WatchSource:0}: Error finding container 2ad74d681897735446021251b3195a2c03071d8cd800023045ce4e769e67d3d5: Status 404 returned error can't find the container with id 2ad74d681897735446021251b3195a2c03071d8cd800023045ce4e769e67d3d5 Oct 30 13:15:26.502023 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:26.501989 4503 generic.go:356] "Generic (PLEG): container finished" podID="6c015946-f285-4a73-ab87-d747da2b3509" containerID="35b69ac95b5d6644eadafb5ee7cae4b84789cd839b7a4524869d34db7fda9d6f" exitCode=0 Oct 30 13:15:26.502434 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:26.502085 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc" event={"ID":"6c015946-f285-4a73-ab87-d747da2b3509","Type":"ContainerDied","Data":"35b69ac95b5d6644eadafb5ee7cae4b84789cd839b7a4524869d34db7fda9d6f"} Oct 30 13:15:26.502434 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:26.502137 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc" event={"ID":"6c015946-f285-4a73-ab87-d747da2b3509","Type":"ContainerStarted","Data":"2ad74d681897735446021251b3195a2c03071d8cd800023045ce4e769e67d3d5"} Oct 30 13:15:27.645827 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:27.645804 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc" Oct 30 13:15:27.784275 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:27.784192 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c015946-f285-4a73-ab87-d747da2b3509-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c015946-f285-4a73-ab87-d747da2b3509-maas-api-xks-ca\") pod \"6c015946-f285-4a73-ab87-d747da2b3509\" (UID: \"6c015946-f285-4a73-ab87-d747da2b3509\") " Oct 30 13:15:27.784275 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:27.784244 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c015946-f285-4a73-ab87-d747da2b3509-kube-api-access-xhn6w\" (UniqueName: \"kubernetes.io/projected/6c015946-f285-4a73-ab87-d747da2b3509-kube-api-access-xhn6w\") pod \"6c015946-f285-4a73-ab87-d747da2b3509\" (UID: \"6c015946-f285-4a73-ab87-d747da2b3509\") " Oct 30 13:15:27.784275 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:27.784278 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c015946-f285-4a73-ab87-d747da2b3509-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c015946-f285-4a73-ab87-d747da2b3509-maas-api-openshift-ca\") pod \"6c015946-f285-4a73-ab87-d747da2b3509\" (UID: \"6c015946-f285-4a73-ab87-d747da2b3509\") " Oct 30 13:15:27.784729 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:27.784698 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c015946-f285-4a73-ab87-d747da2b3509-maas-api-openshift-ca" pod "6c015946-f285-4a73-ab87-d747da2b3509" (UID: "6c015946-f285-4a73-ab87-d747da2b3509"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:27.786393 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:27.786368 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c015946-f285-4a73-ab87-d747da2b3509-kube-api-access-xhn6w" pod "6c015946-f285-4a73-ab87-d747da2b3509" (UID: "6c015946-f285-4a73-ab87-d747da2b3509"). InnerVolumeSpecName "kube-api-access-xhn6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:27.786476 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:27.786392 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c015946-f285-4a73-ab87-d747da2b3509-maas-api-xks-ca" pod "6c015946-f285-4a73-ab87-d747da2b3509" (UID: "6c015946-f285-4a73-ab87-d747da2b3509"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:15:27.885262 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:27.885221 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c015946-f285-4a73-ab87-d747da2b3509-maas-api-xks-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:15:27.885262 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:27.885260 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhn6w\" (UniqueName: \"kubernetes.io/projected/6c015946-f285-4a73-ab87-d747da2b3509-kube-api-access-xhn6w\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:15:27.885262 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:27.885273 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c015946-f285-4a73-ab87-d747da2b3509-maas-api-openshift-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:15:28.515020 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:28.514982 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc" event={"ID":"6c015946-f285-4a73-ab87-d747da2b3509","Type":"ContainerDied","Data":"2ad74d681897735446021251b3195a2c03071d8cd800023045ce4e769e67d3d5"} Oct 30 13:15:28.515020 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:28.515012 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc" Oct 30 13:15:28.515247 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:28.515036 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ad74d681897735446021251b3195a2c03071d8cd800023045ce4e769e67d3d5" Oct 30 13:15:29.699963 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:15:29.699901 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:15:36.209156 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.209121 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6"] Oct 30 13:15:36.209852 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.209828 4503 cpu_manager.go:414] "containerMap: removing container" podUID="6c015946-f285-4a73-ab87-d747da2b3509" containerName="revoke-keys" Oct 30 13:15:36.209852 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.209848 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c015946-f285-4a73-ab87-d747da2b3509" containerName="revoke-keys" Oct 30 13:15:36.209985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.209958 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c015946-f285-4a73-ab87-d747da2b3509" containerName="revoke-keys" Oct 30 13:15:36.209985 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.209973 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d87698-da9f-4805-9037-79a9c4c7b6b6" containerName="cleanup" Oct 30 13:15:36.213976 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.213957 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6" Oct 30 13:15:36.232645 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.232612 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6"] Oct 30 13:15:36.240615 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.240588 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:15:36.240721 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.240618 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:15:36.373246 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.373212 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4rbs\" (UniqueName: \"kubernetes.io/projected/56174a2e-3289-430d-9f76-0161d89bfcde-kube-api-access-s4rbs\") pod \"maas-api-revoke-sub-ec843018-5fct6\" (UID: \"56174a2e-3289-430d-9f76-0161d89bfcde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6" Oct 30 13:15:36.373407 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.373270 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56174a2e-3289-430d-9f76-0161d89bfcde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-5fct6\" (UID: \"56174a2e-3289-430d-9f76-0161d89bfcde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6" Oct 30 13:15:36.373407 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.373354 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56174a2e-3289-430d-9f76-0161d89bfcde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-5fct6\" (UID: \"56174a2e-3289-430d-9f76-0161d89bfcde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6" Oct 30 13:15:36.474971 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.474858 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4rbs\" (UniqueName: \"kubernetes.io/projected/56174a2e-3289-430d-9f76-0161d89bfcde-kube-api-access-s4rbs\") pod \"maas-api-revoke-sub-ec843018-5fct6\" (UID: \"56174a2e-3289-430d-9f76-0161d89bfcde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6" Oct 30 13:15:36.475136 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.474972 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56174a2e-3289-430d-9f76-0161d89bfcde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-5fct6\" (UID: \"56174a2e-3289-430d-9f76-0161d89bfcde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6" Oct 30 13:15:36.475136 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.475037 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56174a2e-3289-430d-9f76-0161d89bfcde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-5fct6\" (UID: \"56174a2e-3289-430d-9f76-0161d89bfcde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6" Oct 30 13:15:36.475717 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.475689 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56174a2e-3289-430d-9f76-0161d89bfcde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-5fct6\" (UID: \"56174a2e-3289-430d-9f76-0161d89bfcde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6" Oct 30 13:15:36.477443 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.477413 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56174a2e-3289-430d-9f76-0161d89bfcde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-5fct6\" (UID: \"56174a2e-3289-430d-9f76-0161d89bfcde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6" Oct 30 13:15:36.499496 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.499461 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4rbs\" (UniqueName: \"kubernetes.io/projected/56174a2e-3289-430d-9f76-0161d89bfcde-kube-api-access-s4rbs\") pod \"maas-api-revoke-sub-ec843018-5fct6\" (UID: \"56174a2e-3289-430d-9f76-0161d89bfcde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6" Oct 30 13:15:36.527358 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.527327 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6" Oct 30 13:15:36.833875 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:36.833850 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6"] Oct 30 13:15:36.835305 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:15:36.835275 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56174a2e_3289_430d_9f76_0161d89bfcde.slice/crio-6a27b9b7564b75b3afe7b8058569c86d3f6f696d2d75eca9d489e6efcfc92c64 WatchSource:0}: Error finding container 6a27b9b7564b75b3afe7b8058569c86d3f6f696d2d75eca9d489e6efcfc92c64: Status 404 returned error can't find the container with id 6a27b9b7564b75b3afe7b8058569c86d3f6f696d2d75eca9d489e6efcfc92c64 Oct 30 13:15:37.566520 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:37.566435 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6" event={"ID":"56174a2e-3289-430d-9f76-0161d89bfcde","Type":"ContainerStarted","Data":"9eed11350005b5e29dbf79272bdbdc1e9cc05176d353976488c24a2c77ca2a71"} Oct 30 13:15:37.566520 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:37.566492 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6" event={"ID":"56174a2e-3289-430d-9f76-0161d89bfcde","Type":"ContainerStarted","Data":"6a27b9b7564b75b3afe7b8058569c86d3f6f696d2d75eca9d489e6efcfc92c64"} Oct 30 13:15:37.612179 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:37.612131 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6" podStartSLOduration=1.6121166759999999 podStartE2EDuration="1.612116676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:37.60952295 +0000 UTC m=+1431.432751227" watchObservedRunningTime="2026-10-30 13:15:37.612116676 +0000 UTC m=+1431.435344954" Oct 30 13:15:38.584310 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:38.584270 4503 generic.go:356] "Generic (PLEG): container finished" podID="56174a2e-3289-430d-9f76-0161d89bfcde" containerID="9eed11350005b5e29dbf79272bdbdc1e9cc05176d353976488c24a2c77ca2a71" exitCode=0 Oct 30 13:15:38.584740 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:38.584372 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6" event={"ID":"56174a2e-3289-430d-9f76-0161d89bfcde","Type":"ContainerDied","Data":"9eed11350005b5e29dbf79272bdbdc1e9cc05176d353976488c24a2c77ca2a71"} Oct 30 13:15:39.801089 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:39.801055 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6" Oct 30 13:15:39.933292 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:39.932369 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56174a2e-3289-430d-9f76-0161d89bfcde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56174a2e-3289-430d-9f76-0161d89bfcde-maas-api-openshift-ca\") pod \"56174a2e-3289-430d-9f76-0161d89bfcde\" (UID: \"56174a2e-3289-430d-9f76-0161d89bfcde\") " Oct 30 13:15:39.933292 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:39.932427 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56174a2e-3289-430d-9f76-0161d89bfcde-kube-api-access-s4rbs\" (UniqueName: \"kubernetes.io/projected/56174a2e-3289-430d-9f76-0161d89bfcde-kube-api-access-s4rbs\") pod \"56174a2e-3289-430d-9f76-0161d89bfcde\" (UID: \"56174a2e-3289-430d-9f76-0161d89bfcde\") " Oct 30 13:15:39.933292 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:39.932483 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56174a2e-3289-430d-9f76-0161d89bfcde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56174a2e-3289-430d-9f76-0161d89bfcde-maas-api-xks-ca\") pod \"56174a2e-3289-430d-9f76-0161d89bfcde\" (UID: \"56174a2e-3289-430d-9f76-0161d89bfcde\") " Oct 30 13:15:39.944468 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:39.944433 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56174a2e-3289-430d-9f76-0161d89bfcde-maas-api-openshift-ca" pod "56174a2e-3289-430d-9f76-0161d89bfcde" (UID: "56174a2e-3289-430d-9f76-0161d89bfcde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:39.958277 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:39.951135 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56174a2e-3289-430d-9f76-0161d89bfcde-kube-api-access-s4rbs" pod "56174a2e-3289-430d-9f76-0161d89bfcde" (UID: "56174a2e-3289-430d-9f76-0161d89bfcde"). InnerVolumeSpecName "kube-api-access-s4rbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:40.053965 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:40.053473 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56174a2e-3289-430d-9f76-0161d89bfcde-maas-api-xks-ca" pod "56174a2e-3289-430d-9f76-0161d89bfcde" (UID: "56174a2e-3289-430d-9f76-0161d89bfcde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:15:40.057960 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:40.054589 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56174a2e-3289-430d-9f76-0161d89bfcde-maas-api-openshift-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:15:40.057960 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:40.054626 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4rbs\" (UniqueName: \"kubernetes.io/projected/56174a2e-3289-430d-9f76-0161d89bfcde-kube-api-access-s4rbs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:15:40.057960 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:40.054644 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56174a2e-3289-430d-9f76-0161d89bfcde-maas-api-xks-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:15:40.613962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:40.613135 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6" event={"ID":"56174a2e-3289-430d-9f76-0161d89bfcde","Type":"ContainerDied","Data":"6a27b9b7564b75b3afe7b8058569c86d3f6f696d2d75eca9d489e6efcfc92c64"} Oct 30 13:15:40.613962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:40.613199 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a27b9b7564b75b3afe7b8058569c86d3f6f696d2d75eca9d489e6efcfc92c64" Oct 30 13:15:40.613962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:15:40.613264 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6" Oct 30 13:15:59.701010 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:15:59.700969 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:16:04.794860 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:04.794815 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc"] Oct 30 13:16:04.795468 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:04.795447 4503 cpu_manager.go:414] "containerMap: removing container" podUID="56174a2e-3289-430d-9f76-0161d89bfcde" containerName="revoke-keys" Oct 30 13:16:04.795554 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:04.795470 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56174a2e-3289-430d-9f76-0161d89bfcde" containerName="revoke-keys" Oct 30 13:16:04.795639 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:04.795624 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="56174a2e-3289-430d-9f76-0161d89bfcde" containerName="revoke-keys" Oct 30 13:16:04.800080 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:04.800059 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc" Oct 30 13:16:04.819859 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:04.819826 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc"] Oct 30 13:16:04.825405 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:04.825379 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6ktb\" (UniqueName: \"kubernetes.io/projected/0bb256e4-67a7-4959-86fa-632133eeeb4e-kube-api-access-q6ktb\") pod \"maas-api-revoke-sub-046e1343-6svcc\" (UID: \"0bb256e4-67a7-4959-86fa-632133eeeb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc" Oct 30 13:16:04.825535 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:04.825412 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb256e4-67a7-4959-86fa-632133eeeb4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-6svcc\" (UID: \"0bb256e4-67a7-4959-86fa-632133eeeb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc" Oct 30 13:16:04.825586 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:04.825536 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb256e4-67a7-4959-86fa-632133eeeb4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-6svcc\" (UID: \"0bb256e4-67a7-4959-86fa-632133eeeb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc" Oct 30 13:16:04.828256 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:04.828236 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:16:04.828320 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:04.828241 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:16:04.927035 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:04.927000 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6ktb\" (UniqueName: \"kubernetes.io/projected/0bb256e4-67a7-4959-86fa-632133eeeb4e-kube-api-access-q6ktb\") pod \"maas-api-revoke-sub-046e1343-6svcc\" (UID: \"0bb256e4-67a7-4959-86fa-632133eeeb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc" Oct 30 13:16:04.927035 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:04.927036 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb256e4-67a7-4959-86fa-632133eeeb4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-6svcc\" (UID: \"0bb256e4-67a7-4959-86fa-632133eeeb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc" Oct 30 13:16:04.927269 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:04.927097 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb256e4-67a7-4959-86fa-632133eeeb4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-6svcc\" (UID: \"0bb256e4-67a7-4959-86fa-632133eeeb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc" Oct 30 13:16:04.927757 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:04.927734 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb256e4-67a7-4959-86fa-632133eeeb4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-6svcc\" (UID: \"0bb256e4-67a7-4959-86fa-632133eeeb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc" Oct 30 13:16:04.929476 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:04.929452 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb256e4-67a7-4959-86fa-632133eeeb4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-6svcc\" (UID: \"0bb256e4-67a7-4959-86fa-632133eeeb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc" Oct 30 13:16:04.951505 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:04.951480 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6ktb\" (UniqueName: \"kubernetes.io/projected/0bb256e4-67a7-4959-86fa-632133eeeb4e-kube-api-access-q6ktb\") pod \"maas-api-revoke-sub-046e1343-6svcc\" (UID: \"0bb256e4-67a7-4959-86fa-632133eeeb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc" Oct 30 13:16:05.112784 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:05.112704 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc" Oct 30 13:16:05.415051 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:05.415021 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc"] Oct 30 13:16:05.416871 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:16:05.416845 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb256e4_67a7_4959_86fa_632133eeeb4e.slice/crio-2f2fcdbb1f432a149777ee93289332629481dd97f674165b121ad405bf762b2d WatchSource:0}: Error finding container 2f2fcdbb1f432a149777ee93289332629481dd97f674165b121ad405bf762b2d: Status 404 returned error can't find the container with id 2f2fcdbb1f432a149777ee93289332629481dd97f674165b121ad405bf762b2d Oct 30 13:16:05.904433 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:05.904393 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc" event={"ID":"0bb256e4-67a7-4959-86fa-632133eeeb4e","Type":"ContainerStarted","Data":"488838be2bf27fe0a391cda8669c0260b4e796d563c23e94365667b74d7b0e9e"} Oct 30 13:16:05.904830 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:05.904491 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc" event={"ID":"0bb256e4-67a7-4959-86fa-632133eeeb4e","Type":"ContainerStarted","Data":"2f2fcdbb1f432a149777ee93289332629481dd97f674165b121ad405bf762b2d"} Oct 30 13:16:05.947284 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:05.947233 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc" podStartSLOduration=1.947219458 podStartE2EDuration="1.947219458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:05.94688238 +0000 UTC m=+1459.770110655" watchObservedRunningTime="2026-10-30 13:16:05.947219458 +0000 UTC m=+1459.770447735" Oct 30 13:16:06.911924 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:06.911886 4503 generic.go:356] "Generic (PLEG): container finished" podID="0bb256e4-67a7-4959-86fa-632133eeeb4e" containerID="488838be2bf27fe0a391cda8669c0260b4e796d563c23e94365667b74d7b0e9e" exitCode=0 Oct 30 13:16:06.912438 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:06.911986 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc" event={"ID":"0bb256e4-67a7-4959-86fa-632133eeeb4e","Type":"ContainerDied","Data":"488838be2bf27fe0a391cda8669c0260b4e796d563c23e94365667b74d7b0e9e"} Oct 30 13:16:08.056184 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:08.056159 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc" Oct 30 13:16:08.159957 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:08.159907 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bb256e4-67a7-4959-86fa-632133eeeb4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb256e4-67a7-4959-86fa-632133eeeb4e-maas-api-openshift-ca\") pod \"0bb256e4-67a7-4959-86fa-632133eeeb4e\" (UID: \"0bb256e4-67a7-4959-86fa-632133eeeb4e\") " Oct 30 13:16:08.160149 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:08.159967 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb256e4-67a7-4959-86fa-632133eeeb4e-kube-api-access-q6ktb\" (UniqueName: \"kubernetes.io/projected/0bb256e4-67a7-4959-86fa-632133eeeb4e-kube-api-access-q6ktb\") pod \"0bb256e4-67a7-4959-86fa-632133eeeb4e\" (UID: \"0bb256e4-67a7-4959-86fa-632133eeeb4e\") " Oct 30 13:16:08.160149 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:08.160114 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bb256e4-67a7-4959-86fa-632133eeeb4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb256e4-67a7-4959-86fa-632133eeeb4e-maas-api-xks-ca\") pod \"0bb256e4-67a7-4959-86fa-632133eeeb4e\" (UID: \"0bb256e4-67a7-4959-86fa-632133eeeb4e\") " Oct 30 13:16:08.160355 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:08.160330 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb256e4-67a7-4959-86fa-632133eeeb4e-maas-api-openshift-ca" pod "0bb256e4-67a7-4959-86fa-632133eeeb4e" (UID: "0bb256e4-67a7-4959-86fa-632133eeeb4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:08.160466 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:08.160451 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb256e4-67a7-4959-86fa-632133eeeb4e-maas-api-openshift-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:16:08.162111 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:08.162083 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb256e4-67a7-4959-86fa-632133eeeb4e-kube-api-access-q6ktb" pod "0bb256e4-67a7-4959-86fa-632133eeeb4e" (UID: "0bb256e4-67a7-4959-86fa-632133eeeb4e"). InnerVolumeSpecName "kube-api-access-q6ktb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:08.162241 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:08.162218 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bb256e4-67a7-4959-86fa-632133eeeb4e-maas-api-xks-ca" pod "0bb256e4-67a7-4959-86fa-632133eeeb4e" (UID: "0bb256e4-67a7-4959-86fa-632133eeeb4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:16:08.261164 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:08.261090 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb256e4-67a7-4959-86fa-632133eeeb4e-maas-api-xks-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:16:08.261164 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:08.261117 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6ktb\" (UniqueName: \"kubernetes.io/projected/0bb256e4-67a7-4959-86fa-632133eeeb4e-kube-api-access-q6ktb\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:16:08.925730 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:08.925690 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc" event={"ID":"0bb256e4-67a7-4959-86fa-632133eeeb4e","Type":"ContainerDied","Data":"2f2fcdbb1f432a149777ee93289332629481dd97f674165b121ad405bf762b2d"} Oct 30 13:16:08.925730 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:08.925740 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f2fcdbb1f432a149777ee93289332629481dd97f674165b121ad405bf762b2d" Oct 30 13:16:08.925978 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:08.925706 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc" Oct 30 13:16:15.183444 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.183410 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8"] Oct 30 13:16:15.184012 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.183991 4503 cpu_manager.go:414] "containerMap: removing container" podUID="0bb256e4-67a7-4959-86fa-632133eeeb4e" containerName="revoke-keys" Oct 30 13:16:15.184072 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.184015 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb256e4-67a7-4959-86fa-632133eeeb4e" containerName="revoke-keys" Oct 30 13:16:15.184152 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.184141 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb256e4-67a7-4959-86fa-632133eeeb4e" containerName="revoke-keys" Oct 30 13:16:15.187620 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.187601 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8" Oct 30 13:16:15.213162 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.213123 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8"] Oct 30 13:16:15.218352 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.218308 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:16:15.218486 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.218444 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:16:15.334771 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.334724 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w28j\" (UniqueName: \"kubernetes.io/projected/6d49097c-84e1-46df-9d5a-a855d38c3831-kube-api-access-5w28j\") pod \"maas-api-revoke-sub-d3fba056-8s9p8\" (UID: \"6d49097c-84e1-46df-9d5a-a855d38c3831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8" Oct 30 13:16:15.334771 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.334776 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d49097c-84e1-46df-9d5a-a855d38c3831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8s9p8\" (UID: \"6d49097c-84e1-46df-9d5a-a855d38c3831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8" Oct 30 13:16:15.335068 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.334968 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d49097c-84e1-46df-9d5a-a855d38c3831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8s9p8\" (UID: \"6d49097c-84e1-46df-9d5a-a855d38c3831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8" Oct 30 13:16:15.436091 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.435995 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d49097c-84e1-46df-9d5a-a855d38c3831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8s9p8\" (UID: \"6d49097c-84e1-46df-9d5a-a855d38c3831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8" Oct 30 13:16:15.436275 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.436103 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w28j\" (UniqueName: \"kubernetes.io/projected/6d49097c-84e1-46df-9d5a-a855d38c3831-kube-api-access-5w28j\") pod \"maas-api-revoke-sub-d3fba056-8s9p8\" (UID: \"6d49097c-84e1-46df-9d5a-a855d38c3831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8" Oct 30 13:16:15.436275 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.436136 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d49097c-84e1-46df-9d5a-a855d38c3831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8s9p8\" (UID: \"6d49097c-84e1-46df-9d5a-a855d38c3831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8" Oct 30 13:16:15.436805 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.436779 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d49097c-84e1-46df-9d5a-a855d38c3831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8s9p8\" (UID: \"6d49097c-84e1-46df-9d5a-a855d38c3831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8" Oct 30 13:16:15.438522 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.438495 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d49097c-84e1-46df-9d5a-a855d38c3831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8s9p8\" (UID: \"6d49097c-84e1-46df-9d5a-a855d38c3831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8" Oct 30 13:16:15.465513 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.465468 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w28j\" (UniqueName: \"kubernetes.io/projected/6d49097c-84e1-46df-9d5a-a855d38c3831-kube-api-access-5w28j\") pod \"maas-api-revoke-sub-d3fba056-8s9p8\" (UID: \"6d49097c-84e1-46df-9d5a-a855d38c3831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8" Oct 30 13:16:15.501678 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.501650 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8" Oct 30 13:16:15.806124 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.806098 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8"] Oct 30 13:16:15.807790 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:16:15.807765 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d49097c_84e1_46df_9d5a_a855d38c3831.slice/crio-703436af3d399c380728d3668cd5fc345b598feeab57583cd0c762af4c6df0ca WatchSource:0}: Error finding container 703436af3d399c380728d3668cd5fc345b598feeab57583cd0c762af4c6df0ca: Status 404 returned error can't find the container with id 703436af3d399c380728d3668cd5fc345b598feeab57583cd0c762af4c6df0ca Oct 30 13:16:15.966665 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:15.966632 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8" event={"ID":"6d49097c-84e1-46df-9d5a-a855d38c3831","Type":"ContainerStarted","Data":"703436af3d399c380728d3668cd5fc345b598feeab57583cd0c762af4c6df0ca"} Oct 30 13:16:16.978180 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:16.978074 4503 generic.go:356] "Generic (PLEG): container finished" podID="6d49097c-84e1-46df-9d5a-a855d38c3831" containerID="51e7647980b7c15d391168d68d827e9d3999674f91f907936bc6cdb494f1cee7" exitCode=0 Oct 30 13:16:16.978668 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:16.978191 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8" event={"ID":"6d49097c-84e1-46df-9d5a-a855d38c3831","Type":"ContainerDied","Data":"51e7647980b7c15d391168d68d827e9d3999674f91f907936bc6cdb494f1cee7"} Oct 30 13:16:18.777964 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:18.775256 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8" Oct 30 13:16:18.909318 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:18.909275 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d49097c-84e1-46df-9d5a-a855d38c3831-kube-api-access-5w28j\" (UniqueName: \"kubernetes.io/projected/6d49097c-84e1-46df-9d5a-a855d38c3831-kube-api-access-5w28j\") pod \"6d49097c-84e1-46df-9d5a-a855d38c3831\" (UID: \"6d49097c-84e1-46df-9d5a-a855d38c3831\") " Oct 30 13:16:18.925259 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:18.916244 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d49097c-84e1-46df-9d5a-a855d38c3831-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d49097c-84e1-46df-9d5a-a855d38c3831-maas-api-openshift-ca\") pod \"6d49097c-84e1-46df-9d5a-a855d38c3831\" (UID: \"6d49097c-84e1-46df-9d5a-a855d38c3831\") " Oct 30 13:16:18.931071 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:18.925213 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d49097c-84e1-46df-9d5a-a855d38c3831-maas-api-openshift-ca" pod "6d49097c-84e1-46df-9d5a-a855d38c3831" (UID: "6d49097c-84e1-46df-9d5a-a855d38c3831"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:18.951584 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:18.931386 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d49097c-84e1-46df-9d5a-a855d38c3831-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d49097c-84e1-46df-9d5a-a855d38c3831-maas-api-xks-ca\") pod \"6d49097c-84e1-46df-9d5a-a855d38c3831\" (UID: \"6d49097c-84e1-46df-9d5a-a855d38c3831\") " Oct 30 13:16:18.982000 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:18.957473 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d49097c-84e1-46df-9d5a-a855d38c3831-maas-api-openshift-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:16:19.086991 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:19.086920 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d49097c-84e1-46df-9d5a-a855d38c3831-kube-api-access-5w28j" pod "6d49097c-84e1-46df-9d5a-a855d38c3831" (UID: "6d49097c-84e1-46df-9d5a-a855d38c3831"). InnerVolumeSpecName "kube-api-access-5w28j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:19.115056 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:19.114988 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8" event={"ID":"6d49097c-84e1-46df-9d5a-a855d38c3831","Type":"ContainerDied","Data":"703436af3d399c380728d3668cd5fc345b598feeab57583cd0c762af4c6df0ca"} Oct 30 13:16:19.115565 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:19.115311 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="703436af3d399c380728d3668cd5fc345b598feeab57583cd0c762af4c6df0ca" Oct 30 13:16:19.115565 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:19.115407 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8" Oct 30 13:16:19.138119 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:19.138079 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d49097c-84e1-46df-9d5a-a855d38c3831-maas-api-xks-ca" pod "6d49097c-84e1-46df-9d5a-a855d38c3831" (UID: "6d49097c-84e1-46df-9d5a-a855d38c3831"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:16:19.197755 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:19.197659 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w28j\" (UniqueName: \"kubernetes.io/projected/6d49097c-84e1-46df-9d5a-a855d38c3831-kube-api-access-5w28j\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:16:19.198009 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:19.197987 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d49097c-84e1-46df-9d5a-a855d38c3831-maas-api-xks-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:16:29.701820 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:16:29.701729 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:16:37.649299 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:37.649261 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr"] Oct 30 13:16:37.649718 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:37.649711 4503 cpu_manager.go:414] "containerMap: removing container" podUID="6d49097c-84e1-46df-9d5a-a855d38c3831" containerName="revoke-keys" Oct 30 13:16:37.649761 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:37.649722 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d49097c-84e1-46df-9d5a-a855d38c3831" containerName="revoke-keys" Oct 30 13:16:37.649802 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:37.649796 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d49097c-84e1-46df-9d5a-a855d38c3831" containerName="revoke-keys" Oct 30 13:16:37.653040 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:37.653012 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr" Oct 30 13:16:37.686026 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:37.685993 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:16:37.695352 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:37.695325 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:16:37.710956 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:37.710921 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr"] Oct 30 13:16:37.746919 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:37.746891 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gxnd\" (UniqueName: \"kubernetes.io/projected/5388e3b6-9852-4c65-a53f-eb03413fe27d-kube-api-access-5gxnd\") pod \"maas-api-revoke-sub-ae594979-7phhr\" (UID: \"5388e3b6-9852-4c65-a53f-eb03413fe27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr" Oct 30 13:16:37.747093 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:37.746928 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5388e3b6-9852-4c65-a53f-eb03413fe27d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-7phhr\" (UID: \"5388e3b6-9852-4c65-a53f-eb03413fe27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr" Oct 30 13:16:37.747206 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:37.747180 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5388e3b6-9852-4c65-a53f-eb03413fe27d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-7phhr\" (UID: \"5388e3b6-9852-4c65-a53f-eb03413fe27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr" Oct 30 13:16:37.847860 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:37.847821 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5388e3b6-9852-4c65-a53f-eb03413fe27d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-7phhr\" (UID: \"5388e3b6-9852-4c65-a53f-eb03413fe27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr" Oct 30 13:16:37.848074 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:37.847901 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gxnd\" (UniqueName: \"kubernetes.io/projected/5388e3b6-9852-4c65-a53f-eb03413fe27d-kube-api-access-5gxnd\") pod \"maas-api-revoke-sub-ae594979-7phhr\" (UID: \"5388e3b6-9852-4c65-a53f-eb03413fe27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr" Oct 30 13:16:37.848074 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:37.847960 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5388e3b6-9852-4c65-a53f-eb03413fe27d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-7phhr\" (UID: \"5388e3b6-9852-4c65-a53f-eb03413fe27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr" Oct 30 13:16:37.848602 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:37.848580 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5388e3b6-9852-4c65-a53f-eb03413fe27d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-7phhr\" (UID: \"5388e3b6-9852-4c65-a53f-eb03413fe27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr" Oct 30 13:16:37.850312 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:37.850293 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5388e3b6-9852-4c65-a53f-eb03413fe27d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-7phhr\" (UID: \"5388e3b6-9852-4c65-a53f-eb03413fe27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr" Oct 30 13:16:37.875248 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:37.875216 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gxnd\" (UniqueName: \"kubernetes.io/projected/5388e3b6-9852-4c65-a53f-eb03413fe27d-kube-api-access-5gxnd\") pod \"maas-api-revoke-sub-ae594979-7phhr\" (UID: \"5388e3b6-9852-4c65-a53f-eb03413fe27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr" Oct 30 13:16:37.966569 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:37.966491 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr" Oct 30 13:16:38.267914 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:38.267888 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr"] Oct 30 13:16:38.269559 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:16:38.269517 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5388e3b6_9852_4c65_a53f_eb03413fe27d.slice/crio-8e73b96f6ed6cd64ce8cd9dd79139836f323228773cb610ccafeb3a8a3991ceb WatchSource:0}: Error finding container 8e73b96f6ed6cd64ce8cd9dd79139836f323228773cb610ccafeb3a8a3991ceb: Status 404 returned error can't find the container with id 8e73b96f6ed6cd64ce8cd9dd79139836f323228773cb610ccafeb3a8a3991ceb Oct 30 13:16:38.291813 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:38.291786 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr" event={"ID":"5388e3b6-9852-4c65-a53f-eb03413fe27d","Type":"ContainerStarted","Data":"8e73b96f6ed6cd64ce8cd9dd79139836f323228773cb610ccafeb3a8a3991ceb"} Oct 30 13:16:39.298753 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:39.298714 4503 generic.go:356] "Generic (PLEG): container finished" podID="5388e3b6-9852-4c65-a53f-eb03413fe27d" containerID="bdb869baa937b324e11e3793e3bac6450d2681fa5e8794a08f169a571df42f00" exitCode=0 Oct 30 13:16:39.299267 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:39.298765 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr" event={"ID":"5388e3b6-9852-4c65-a53f-eb03413fe27d","Type":"ContainerDied","Data":"bdb869baa937b324e11e3793e3bac6450d2681fa5e8794a08f169a571df42f00"} Oct 30 13:16:40.432725 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:40.432701 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr" Oct 30 13:16:40.575463 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:40.575364 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5388e3b6-9852-4c65-a53f-eb03413fe27d-kube-api-access-5gxnd\" (UniqueName: \"kubernetes.io/projected/5388e3b6-9852-4c65-a53f-eb03413fe27d-kube-api-access-5gxnd\") pod \"5388e3b6-9852-4c65-a53f-eb03413fe27d\" (UID: \"5388e3b6-9852-4c65-a53f-eb03413fe27d\") " Oct 30 13:16:40.575463 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:40.575421 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5388e3b6-9852-4c65-a53f-eb03413fe27d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5388e3b6-9852-4c65-a53f-eb03413fe27d-maas-api-xks-ca\") pod \"5388e3b6-9852-4c65-a53f-eb03413fe27d\" (UID: \"5388e3b6-9852-4c65-a53f-eb03413fe27d\") " Oct 30 13:16:40.575679 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:40.575551 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5388e3b6-9852-4c65-a53f-eb03413fe27d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5388e3b6-9852-4c65-a53f-eb03413fe27d-maas-api-openshift-ca\") pod \"5388e3b6-9852-4c65-a53f-eb03413fe27d\" (UID: \"5388e3b6-9852-4c65-a53f-eb03413fe27d\") " Oct 30 13:16:40.575903 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:40.575877 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5388e3b6-9852-4c65-a53f-eb03413fe27d-maas-api-openshift-ca" pod "5388e3b6-9852-4c65-a53f-eb03413fe27d" (UID: "5388e3b6-9852-4c65-a53f-eb03413fe27d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:40.577526 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:40.577500 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5388e3b6-9852-4c65-a53f-eb03413fe27d-maas-api-xks-ca" pod "5388e3b6-9852-4c65-a53f-eb03413fe27d" (UID: "5388e3b6-9852-4c65-a53f-eb03413fe27d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:16:40.577631 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:40.577597 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5388e3b6-9852-4c65-a53f-eb03413fe27d-kube-api-access-5gxnd" pod "5388e3b6-9852-4c65-a53f-eb03413fe27d" (UID: "5388e3b6-9852-4c65-a53f-eb03413fe27d"). InnerVolumeSpecName "kube-api-access-5gxnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:40.677336 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:40.677299 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gxnd\" (UniqueName: \"kubernetes.io/projected/5388e3b6-9852-4c65-a53f-eb03413fe27d-kube-api-access-5gxnd\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:16:40.677336 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:40.677332 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5388e3b6-9852-4c65-a53f-eb03413fe27d-maas-api-xks-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:16:40.677336 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:40.677345 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5388e3b6-9852-4c65-a53f-eb03413fe27d-maas-api-openshift-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:16:41.311925 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:41.311890 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr" event={"ID":"5388e3b6-9852-4c65-a53f-eb03413fe27d","Type":"ContainerDied","Data":"8e73b96f6ed6cd64ce8cd9dd79139836f323228773cb610ccafeb3a8a3991ceb"} Oct 30 13:16:41.312107 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:41.311899 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr" Oct 30 13:16:41.312107 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:41.311959 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e73b96f6ed6cd64ce8cd9dd79139836f323228773cb610ccafeb3a8a3991ceb" Oct 30 13:16:47.748428 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:47.748391 4503 scope.go:122] "RemoveContainer" containerID="b71c4743a919c38ffd47173db043f957593e20052b590a28054cf60977388916" Oct 30 13:16:58.173577 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:58.173540 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6"] Oct 30 13:16:58.174264 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:58.174241 4503 cpu_manager.go:414] "containerMap: removing container" podUID="5388e3b6-9852-4c65-a53f-eb03413fe27d" containerName="revoke-keys" Oct 30 13:16:58.174327 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:58.174267 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5388e3b6-9852-4c65-a53f-eb03413fe27d" containerName="revoke-keys" Oct 30 13:16:58.174392 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:58.174381 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="5388e3b6-9852-4c65-a53f-eb03413fe27d" containerName="revoke-keys" Oct 30 13:16:58.179001 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:58.178981 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6" Oct 30 13:16:58.198428 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:58.198372 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6"] Oct 30 13:16:58.277832 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:58.277797 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:16:58.278027 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:58.277982 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:16:58.283508 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:58.283482 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfhxc\" (UniqueName: \"kubernetes.io/projected/2cc27019-ab9f-4475-bfc1-8d52bc737e62-kube-api-access-lfhxc\") pod \"maas-api-revoke-sub-a4aef7d7-cpbq6\" (UID: \"2cc27019-ab9f-4475-bfc1-8d52bc737e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6" Oct 30 13:16:58.283582 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:58.283522 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc27019-ab9f-4475-bfc1-8d52bc737e62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cpbq6\" (UID: \"2cc27019-ab9f-4475-bfc1-8d52bc737e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6" Oct 30 13:16:58.283705 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:58.283690 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc27019-ab9f-4475-bfc1-8d52bc737e62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cpbq6\" (UID: \"2cc27019-ab9f-4475-bfc1-8d52bc737e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6" Oct 30 13:16:58.385292 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:58.385251 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc27019-ab9f-4475-bfc1-8d52bc737e62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cpbq6\" (UID: \"2cc27019-ab9f-4475-bfc1-8d52bc737e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6" Oct 30 13:16:58.385491 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:58.385336 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfhxc\" (UniqueName: \"kubernetes.io/projected/2cc27019-ab9f-4475-bfc1-8d52bc737e62-kube-api-access-lfhxc\") pod \"maas-api-revoke-sub-a4aef7d7-cpbq6\" (UID: \"2cc27019-ab9f-4475-bfc1-8d52bc737e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6" Oct 30 13:16:58.385491 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:58.385365 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc27019-ab9f-4475-bfc1-8d52bc737e62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cpbq6\" (UID: \"2cc27019-ab9f-4475-bfc1-8d52bc737e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6" Oct 30 13:16:58.385952 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:58.385913 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc27019-ab9f-4475-bfc1-8d52bc737e62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cpbq6\" (UID: \"2cc27019-ab9f-4475-bfc1-8d52bc737e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6" Oct 30 13:16:58.387715 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:58.387693 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc27019-ab9f-4475-bfc1-8d52bc737e62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cpbq6\" (UID: \"2cc27019-ab9f-4475-bfc1-8d52bc737e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6" Oct 30 13:16:58.410883 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:58.410854 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfhxc\" (UniqueName: \"kubernetes.io/projected/2cc27019-ab9f-4475-bfc1-8d52bc737e62-kube-api-access-lfhxc\") pod \"maas-api-revoke-sub-a4aef7d7-cpbq6\" (UID: \"2cc27019-ab9f-4475-bfc1-8d52bc737e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6" Oct 30 13:16:58.492021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:58.491923 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6" Oct 30 13:16:58.803316 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:58.803287 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6"] Oct 30 13:16:58.804847 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:16:58.804819 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc27019_ab9f_4475_bfc1_8d52bc737e62.slice/crio-8ffee4f6a09e10dbc6c10a744c7694af73314e17bb586fe944ff0575e80505de WatchSource:0}: Error finding container 8ffee4f6a09e10dbc6c10a744c7694af73314e17bb586fe944ff0575e80505de: Status 404 returned error can't find the container with id 8ffee4f6a09e10dbc6c10a744c7694af73314e17bb586fe944ff0575e80505de Oct 30 13:16:59.472296 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:59.472258 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6" event={"ID":"2cc27019-ab9f-4475-bfc1-8d52bc737e62","Type":"ContainerStarted","Data":"25b846dca2bf694160e50d5d67eac84fa0e3d2e9a4281cc5160d5f314cb1b7be"} Oct 30 13:16:59.472692 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:59.472338 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6" event={"ID":"2cc27019-ab9f-4475-bfc1-8d52bc737e62","Type":"ContainerStarted","Data":"8ffee4f6a09e10dbc6c10a744c7694af73314e17bb586fe944ff0575e80505de"} Oct 30 13:16:59.514239 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:16:59.514188 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6" podStartSLOduration=1.514175089 podStartE2EDuration="1.514175089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:59.51404719 +0000 UTC m=+1513.337275467" watchObservedRunningTime="2026-10-30 13:16:59.514175089 +0000 UTC m=+1513.337403367" Oct 30 13:16:59.676566 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:16:59.676523 4503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc27019_ab9f_4475_bfc1_8d52bc737e62.slice/crio-25b846dca2bf694160e50d5d67eac84fa0e3d2e9a4281cc5160d5f314cb1b7be.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc27019_ab9f_4475_bfc1_8d52bc737e62.slice/crio-conmon-25b846dca2bf694160e50d5d67eac84fa0e3d2e9a4281cc5160d5f314cb1b7be.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:16:59.700358 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:16:59.700320 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:17:00.479541 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:00.479501 4503 generic.go:356] "Generic (PLEG): container finished" podID="2cc27019-ab9f-4475-bfc1-8d52bc737e62" containerID="25b846dca2bf694160e50d5d67eac84fa0e3d2e9a4281cc5160d5f314cb1b7be" exitCode=0 Oct 30 13:17:00.479984 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:00.479572 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6" event={"ID":"2cc27019-ab9f-4475-bfc1-8d52bc737e62","Type":"ContainerDied","Data":"25b846dca2bf694160e50d5d67eac84fa0e3d2e9a4281cc5160d5f314cb1b7be"} Oct 30 13:17:01.633434 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:01.633407 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6" Oct 30 13:17:01.823747 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:01.823652 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cc27019-ab9f-4475-bfc1-8d52bc737e62-kube-api-access-lfhxc\" (UniqueName: \"kubernetes.io/projected/2cc27019-ab9f-4475-bfc1-8d52bc737e62-kube-api-access-lfhxc\") pod \"2cc27019-ab9f-4475-bfc1-8d52bc737e62\" (UID: \"2cc27019-ab9f-4475-bfc1-8d52bc737e62\") " Oct 30 13:17:01.823747 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:01.823697 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cc27019-ab9f-4475-bfc1-8d52bc737e62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc27019-ab9f-4475-bfc1-8d52bc737e62-maas-api-xks-ca\") pod \"2cc27019-ab9f-4475-bfc1-8d52bc737e62\" (UID: \"2cc27019-ab9f-4475-bfc1-8d52bc737e62\") " Oct 30 13:17:01.823747 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:01.823739 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cc27019-ab9f-4475-bfc1-8d52bc737e62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc27019-ab9f-4475-bfc1-8d52bc737e62-maas-api-openshift-ca\") pod \"2cc27019-ab9f-4475-bfc1-8d52bc737e62\" (UID: \"2cc27019-ab9f-4475-bfc1-8d52bc737e62\") " Oct 30 13:17:01.824238 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:01.824216 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc27019-ab9f-4475-bfc1-8d52bc737e62-maas-api-openshift-ca" pod "2cc27019-ab9f-4475-bfc1-8d52bc737e62" (UID: "2cc27019-ab9f-4475-bfc1-8d52bc737e62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:01.825830 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:01.825807 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc27019-ab9f-4475-bfc1-8d52bc737e62-kube-api-access-lfhxc" pod "2cc27019-ab9f-4475-bfc1-8d52bc737e62" (UID: "2cc27019-ab9f-4475-bfc1-8d52bc737e62"). InnerVolumeSpecName "kube-api-access-lfhxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:01.825830 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:01.825813 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cc27019-ab9f-4475-bfc1-8d52bc737e62-maas-api-xks-ca" pod "2cc27019-ab9f-4475-bfc1-8d52bc737e62" (UID: "2cc27019-ab9f-4475-bfc1-8d52bc737e62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:17:01.925304 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:01.925264 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfhxc\" (UniqueName: \"kubernetes.io/projected/2cc27019-ab9f-4475-bfc1-8d52bc737e62-kube-api-access-lfhxc\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:17:01.925304 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:01.925299 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc27019-ab9f-4475-bfc1-8d52bc737e62-maas-api-xks-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:17:01.925304 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:01.925312 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc27019-ab9f-4475-bfc1-8d52bc737e62-maas-api-openshift-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:17:02.492322 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:02.492295 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6" Oct 30 13:17:02.492492 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:02.492294 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6" event={"ID":"2cc27019-ab9f-4475-bfc1-8d52bc737e62","Type":"ContainerDied","Data":"8ffee4f6a09e10dbc6c10a744c7694af73314e17bb586fe944ff0575e80505de"} Oct 30 13:17:02.492492 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:02.492420 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ffee4f6a09e10dbc6c10a744c7694af73314e17bb586fe944ff0575e80505de" Oct 30 13:17:18.220530 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:18.220494 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd"] Oct 30 13:17:18.221076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:18.220913 4503 cpu_manager.go:414] "containerMap: removing container" podUID="2cc27019-ab9f-4475-bfc1-8d52bc737e62" containerName="revoke-keys" Oct 30 13:17:18.221076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:18.220925 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc27019-ab9f-4475-bfc1-8d52bc737e62" containerName="revoke-keys" Oct 30 13:17:18.221076 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:18.221038 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc27019-ab9f-4475-bfc1-8d52bc737e62" containerName="revoke-keys" Oct 30 13:17:18.224420 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:18.224395 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd" Oct 30 13:17:18.237793 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:18.237763 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd"] Oct 30 13:17:18.255977 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:18.255952 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:17:18.256095 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:18.256008 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:17:18.371507 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:18.371478 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/035c0081-10fb-487e-8881-dbd7f7a6aa33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-j4hcd\" (UID: \"035c0081-10fb-487e-8881-dbd7f7a6aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd" Oct 30 13:17:18.371688 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:18.371549 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/035c0081-10fb-487e-8881-dbd7f7a6aa33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-j4hcd\" (UID: \"035c0081-10fb-487e-8881-dbd7f7a6aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd" Oct 30 13:17:18.371688 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:18.371597 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hgdf\" (UniqueName: \"kubernetes.io/projected/035c0081-10fb-487e-8881-dbd7f7a6aa33-kube-api-access-6hgdf\") pod \"maas-api-revoke-sub-79805ca9-j4hcd\" (UID: \"035c0081-10fb-487e-8881-dbd7f7a6aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd" Oct 30 13:17:18.472886 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:18.472803 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/035c0081-10fb-487e-8881-dbd7f7a6aa33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-j4hcd\" (UID: \"035c0081-10fb-487e-8881-dbd7f7a6aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd" Oct 30 13:17:18.472886 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:18.472862 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hgdf\" (UniqueName: \"kubernetes.io/projected/035c0081-10fb-487e-8881-dbd7f7a6aa33-kube-api-access-6hgdf\") pod \"maas-api-revoke-sub-79805ca9-j4hcd\" (UID: \"035c0081-10fb-487e-8881-dbd7f7a6aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd" Oct 30 13:17:18.473143 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:18.473080 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/035c0081-10fb-487e-8881-dbd7f7a6aa33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-j4hcd\" (UID: \"035c0081-10fb-487e-8881-dbd7f7a6aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd" Oct 30 13:17:18.473668 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:18.473637 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/035c0081-10fb-487e-8881-dbd7f7a6aa33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-j4hcd\" (UID: \"035c0081-10fb-487e-8881-dbd7f7a6aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd" Oct 30 13:17:18.475434 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:18.475415 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/035c0081-10fb-487e-8881-dbd7f7a6aa33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-j4hcd\" (UID: \"035c0081-10fb-487e-8881-dbd7f7a6aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd" Oct 30 13:17:18.498910 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:18.498879 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hgdf\" (UniqueName: \"kubernetes.io/projected/035c0081-10fb-487e-8881-dbd7f7a6aa33-kube-api-access-6hgdf\") pod \"maas-api-revoke-sub-79805ca9-j4hcd\" (UID: \"035c0081-10fb-487e-8881-dbd7f7a6aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd" Oct 30 13:17:18.538481 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:18.538451 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd" Oct 30 13:17:18.844581 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:18.844554 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd"] Oct 30 13:17:18.845709 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:17:18.845678 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod035c0081_10fb_487e_8881_dbd7f7a6aa33.slice/crio-f945f349d57541d18843b54f6e8e857158a7998a16235d4ad470b0dfe3cb3dff WatchSource:0}: Error finding container f945f349d57541d18843b54f6e8e857158a7998a16235d4ad470b0dfe3cb3dff: Status 404 returned error can't find the container with id f945f349d57541d18843b54f6e8e857158a7998a16235d4ad470b0dfe3cb3dff Oct 30 13:17:19.620566 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:19.620477 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd" event={"ID":"035c0081-10fb-487e-8881-dbd7f7a6aa33","Type":"ContainerStarted","Data":"5444117418b9225a470c9c3010cbb8df9d087ec062ed628ba56d6ab558cdc500"} Oct 30 13:17:19.620566 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:19.620542 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd" event={"ID":"035c0081-10fb-487e-8881-dbd7f7a6aa33","Type":"ContainerStarted","Data":"f945f349d57541d18843b54f6e8e857158a7998a16235d4ad470b0dfe3cb3dff"} Oct 30 13:17:19.664311 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:19.664264 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd" podStartSLOduration=1.6642491179999999 podStartE2EDuration="1.664249118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:19.661459457 +0000 UTC m=+1533.484687734" watchObservedRunningTime="2026-10-30 13:17:19.664249118 +0000 UTC m=+1533.487477394" Oct 30 13:17:20.627753 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:20.627706 4503 generic.go:356] "Generic (PLEG): container finished" podID="035c0081-10fb-487e-8881-dbd7f7a6aa33" containerID="5444117418b9225a470c9c3010cbb8df9d087ec062ed628ba56d6ab558cdc500" exitCode=0 Oct 30 13:17:20.628167 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:20.627759 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd" event={"ID":"035c0081-10fb-487e-8881-dbd7f7a6aa33","Type":"ContainerDied","Data":"5444117418b9225a470c9c3010cbb8df9d087ec062ed628ba56d6ab558cdc500"} Oct 30 13:17:21.783368 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:21.783343 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd" Oct 30 13:17:21.908300 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:21.908267 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/035c0081-10fb-487e-8881-dbd7f7a6aa33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/035c0081-10fb-487e-8881-dbd7f7a6aa33-maas-api-xks-ca\") pod \"035c0081-10fb-487e-8881-dbd7f7a6aa33\" (UID: \"035c0081-10fb-487e-8881-dbd7f7a6aa33\") " Oct 30 13:17:21.908480 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:21.908371 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/035c0081-10fb-487e-8881-dbd7f7a6aa33-kube-api-access-6hgdf\" (UniqueName: \"kubernetes.io/projected/035c0081-10fb-487e-8881-dbd7f7a6aa33-kube-api-access-6hgdf\") pod \"035c0081-10fb-487e-8881-dbd7f7a6aa33\" (UID: \"035c0081-10fb-487e-8881-dbd7f7a6aa33\") " Oct 30 13:17:21.908480 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:21.908410 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/035c0081-10fb-487e-8881-dbd7f7a6aa33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/035c0081-10fb-487e-8881-dbd7f7a6aa33-maas-api-openshift-ca\") pod \"035c0081-10fb-487e-8881-dbd7f7a6aa33\" (UID: \"035c0081-10fb-487e-8881-dbd7f7a6aa33\") " Oct 30 13:17:21.908831 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:21.908801 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/035c0081-10fb-487e-8881-dbd7f7a6aa33-maas-api-openshift-ca" pod "035c0081-10fb-487e-8881-dbd7f7a6aa33" (UID: "035c0081-10fb-487e-8881-dbd7f7a6aa33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:21.910551 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:21.910516 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/035c0081-10fb-487e-8881-dbd7f7a6aa33-maas-api-xks-ca" pod "035c0081-10fb-487e-8881-dbd7f7a6aa33" (UID: "035c0081-10fb-487e-8881-dbd7f7a6aa33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:17:21.910653 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:21.910578 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/035c0081-10fb-487e-8881-dbd7f7a6aa33-kube-api-access-6hgdf" pod "035c0081-10fb-487e-8881-dbd7f7a6aa33" (UID: "035c0081-10fb-487e-8881-dbd7f7a6aa33"). InnerVolumeSpecName "kube-api-access-6hgdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:22.009375 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:22.009338 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/035c0081-10fb-487e-8881-dbd7f7a6aa33-maas-api-xks-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:17:22.009375 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:22.009367 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hgdf\" (UniqueName: \"kubernetes.io/projected/035c0081-10fb-487e-8881-dbd7f7a6aa33-kube-api-access-6hgdf\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:17:22.009375 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:22.009378 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/035c0081-10fb-487e-8881-dbd7f7a6aa33-maas-api-openshift-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:17:22.640421 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:22.640338 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd" event={"ID":"035c0081-10fb-487e-8881-dbd7f7a6aa33","Type":"ContainerDied","Data":"f945f349d57541d18843b54f6e8e857158a7998a16235d4ad470b0dfe3cb3dff"} Oct 30 13:17:22.640421 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:22.640359 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd" Oct 30 13:17:22.640421 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:22.640387 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f945f349d57541d18843b54f6e8e857158a7998a16235d4ad470b0dfe3cb3dff" Oct 30 13:17:29.701120 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:17:29.701075 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:17:35.021431 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.021394 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs"] Oct 30 13:17:35.022141 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.022119 4503 cpu_manager.go:414] "containerMap: removing container" podUID="035c0081-10fb-487e-8881-dbd7f7a6aa33" containerName="revoke-keys" Oct 30 13:17:35.022200 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.022144 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="035c0081-10fb-487e-8881-dbd7f7a6aa33" containerName="revoke-keys" Oct 30 13:17:35.022308 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.022294 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="035c0081-10fb-487e-8881-dbd7f7a6aa33" containerName="revoke-keys" Oct 30 13:17:35.026087 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.026062 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs" Oct 30 13:17:35.045272 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.045231 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs"] Oct 30 13:17:35.053413 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.053382 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:17:35.053649 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.053627 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:17:35.094358 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.094319 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a510b36a-5ce1-4047-8fcb-39d341e880c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-v5mgs\" (UID: \"a510b36a-5ce1-4047-8fcb-39d341e880c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs" Oct 30 13:17:35.094579 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.094386 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a510b36a-5ce1-4047-8fcb-39d341e880c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-v5mgs\" (UID: \"a510b36a-5ce1-4047-8fcb-39d341e880c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs" Oct 30 13:17:35.094657 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.094583 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlkz6\" (UniqueName: \"kubernetes.io/projected/a510b36a-5ce1-4047-8fcb-39d341e880c8-kube-api-access-nlkz6\") pod \"maas-api-revoke-sub-0b7e8edb-v5mgs\" (UID: \"a510b36a-5ce1-4047-8fcb-39d341e880c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs" Oct 30 13:17:35.195546 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.195505 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a510b36a-5ce1-4047-8fcb-39d341e880c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-v5mgs\" (UID: \"a510b36a-5ce1-4047-8fcb-39d341e880c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs" Oct 30 13:17:35.195760 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.195568 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a510b36a-5ce1-4047-8fcb-39d341e880c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-v5mgs\" (UID: \"a510b36a-5ce1-4047-8fcb-39d341e880c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs" Oct 30 13:17:35.195760 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.195681 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlkz6\" (UniqueName: \"kubernetes.io/projected/a510b36a-5ce1-4047-8fcb-39d341e880c8-kube-api-access-nlkz6\") pod \"maas-api-revoke-sub-0b7e8edb-v5mgs\" (UID: \"a510b36a-5ce1-4047-8fcb-39d341e880c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs" Oct 30 13:17:35.196555 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.196522 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a510b36a-5ce1-4047-8fcb-39d341e880c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-v5mgs\" (UID: \"a510b36a-5ce1-4047-8fcb-39d341e880c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs" Oct 30 13:17:35.198083 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.198061 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a510b36a-5ce1-4047-8fcb-39d341e880c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-v5mgs\" (UID: \"a510b36a-5ce1-4047-8fcb-39d341e880c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs" Oct 30 13:17:35.223030 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.223003 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlkz6\" (UniqueName: \"kubernetes.io/projected/a510b36a-5ce1-4047-8fcb-39d341e880c8-kube-api-access-nlkz6\") pod \"maas-api-revoke-sub-0b7e8edb-v5mgs\" (UID: \"a510b36a-5ce1-4047-8fcb-39d341e880c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs" Oct 30 13:17:35.340424 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.340336 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs" Oct 30 13:17:35.647083 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.647050 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs"] Oct 30 13:17:35.648635 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:17:35.648606 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda510b36a_5ce1_4047_8fcb_39d341e880c8.slice/crio-353712bb22dd1ee02dcea6ef888c5a4f6b7ece858686aca259498b7405131fc8 WatchSource:0}: Error finding container 353712bb22dd1ee02dcea6ef888c5a4f6b7ece858686aca259498b7405131fc8: Status 404 returned error can't find the container with id 353712bb22dd1ee02dcea6ef888c5a4f6b7ece858686aca259498b7405131fc8 Oct 30 13:17:35.772921 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:35.772885 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs" event={"ID":"a510b36a-5ce1-4047-8fcb-39d341e880c8","Type":"ContainerStarted","Data":"353712bb22dd1ee02dcea6ef888c5a4f6b7ece858686aca259498b7405131fc8"} Oct 30 13:17:36.780379 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:36.780285 4503 generic.go:356] "Generic (PLEG): container finished" podID="a510b36a-5ce1-4047-8fcb-39d341e880c8" containerID="bea88ae43fb4cb3e11a018d677ae4c9fe32b5631ade147bb73782549c13d0391" exitCode=0 Oct 30 13:17:36.780751 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:36.780385 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs" event={"ID":"a510b36a-5ce1-4047-8fcb-39d341e880c8","Type":"ContainerDied","Data":"bea88ae43fb4cb3e11a018d677ae4c9fe32b5631ade147bb73782549c13d0391"} Oct 30 13:17:37.928490 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:37.928467 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs" Oct 30 13:17:38.024507 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:38.024472 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a510b36a-5ce1-4047-8fcb-39d341e880c8-kube-api-access-nlkz6\" (UniqueName: \"kubernetes.io/projected/a510b36a-5ce1-4047-8fcb-39d341e880c8-kube-api-access-nlkz6\") pod \"a510b36a-5ce1-4047-8fcb-39d341e880c8\" (UID: \"a510b36a-5ce1-4047-8fcb-39d341e880c8\") " Oct 30 13:17:38.024688 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:38.024580 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a510b36a-5ce1-4047-8fcb-39d341e880c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a510b36a-5ce1-4047-8fcb-39d341e880c8-maas-api-openshift-ca\") pod \"a510b36a-5ce1-4047-8fcb-39d341e880c8\" (UID: \"a510b36a-5ce1-4047-8fcb-39d341e880c8\") " Oct 30 13:17:38.024688 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:38.024669 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a510b36a-5ce1-4047-8fcb-39d341e880c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a510b36a-5ce1-4047-8fcb-39d341e880c8-maas-api-xks-ca\") pod \"a510b36a-5ce1-4047-8fcb-39d341e880c8\" (UID: \"a510b36a-5ce1-4047-8fcb-39d341e880c8\") " Oct 30 13:17:38.025041 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:38.025016 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a510b36a-5ce1-4047-8fcb-39d341e880c8-maas-api-openshift-ca" pod "a510b36a-5ce1-4047-8fcb-39d341e880c8" (UID: "a510b36a-5ce1-4047-8fcb-39d341e880c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:38.026674 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:38.026649 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a510b36a-5ce1-4047-8fcb-39d341e880c8-maas-api-xks-ca" pod "a510b36a-5ce1-4047-8fcb-39d341e880c8" (UID: "a510b36a-5ce1-4047-8fcb-39d341e880c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:17:38.026792 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:38.026703 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a510b36a-5ce1-4047-8fcb-39d341e880c8-kube-api-access-nlkz6" pod "a510b36a-5ce1-4047-8fcb-39d341e880c8" (UID: "a510b36a-5ce1-4047-8fcb-39d341e880c8"). InnerVolumeSpecName "kube-api-access-nlkz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:38.126025 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:38.125898 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a510b36a-5ce1-4047-8fcb-39d341e880c8-maas-api-openshift-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:17:38.126025 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:38.125933 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a510b36a-5ce1-4047-8fcb-39d341e880c8-maas-api-xks-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:17:38.126025 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:38.125970 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlkz6\" (UniqueName: \"kubernetes.io/projected/a510b36a-5ce1-4047-8fcb-39d341e880c8-kube-api-access-nlkz6\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:17:38.793025 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:38.792989 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs" event={"ID":"a510b36a-5ce1-4047-8fcb-39d341e880c8","Type":"ContainerDied","Data":"353712bb22dd1ee02dcea6ef888c5a4f6b7ece858686aca259498b7405131fc8"} Oct 30 13:17:38.793197 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:38.793042 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="353712bb22dd1ee02dcea6ef888c5a4f6b7ece858686aca259498b7405131fc8" Oct 30 13:17:38.793197 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:38.793007 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs" Oct 30 13:17:45.879867 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:45.879822 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q"] Oct 30 13:17:45.880395 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:45.880345 4503 cpu_manager.go:414] "containerMap: removing container" podUID="a510b36a-5ce1-4047-8fcb-39d341e880c8" containerName="revoke-keys" Oct 30 13:17:45.880395 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:45.880361 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a510b36a-5ce1-4047-8fcb-39d341e880c8" containerName="revoke-keys" Oct 30 13:17:45.880519 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:45.880433 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="a510b36a-5ce1-4047-8fcb-39d341e880c8" containerName="revoke-keys" Oct 30 13:17:45.883967 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:45.883929 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q" Oct 30 13:17:45.907682 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:45.907648 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q"] Oct 30 13:17:45.915236 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:45.915199 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:17:45.915404 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:45.915302 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:17:46.009832 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:46.009794 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-k6p6q\" (UID: \"6e7dd5b1-9792-4bee-9b5c-f934e2089c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q" Oct 30 13:17:46.010082 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:46.009998 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-k6p6q\" (UID: \"6e7dd5b1-9792-4bee-9b5c-f934e2089c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q" Oct 30 13:17:46.010158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:46.010076 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxzwz\" (UniqueName: \"kubernetes.io/projected/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-kube-api-access-xxzwz\") pod \"maas-api-revoke-sub-3d4156f1-k6p6q\" (UID: \"6e7dd5b1-9792-4bee-9b5c-f934e2089c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q" Oct 30 13:17:46.111553 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:46.111514 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-k6p6q\" (UID: \"6e7dd5b1-9792-4bee-9b5c-f934e2089c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q" Oct 30 13:17:46.111757 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:46.111629 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-k6p6q\" (UID: \"6e7dd5b1-9792-4bee-9b5c-f934e2089c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q" Oct 30 13:17:46.111757 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:46.111670 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxzwz\" (UniqueName: \"kubernetes.io/projected/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-kube-api-access-xxzwz\") pod \"maas-api-revoke-sub-3d4156f1-k6p6q\" (UID: \"6e7dd5b1-9792-4bee-9b5c-f934e2089c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q" Oct 30 13:17:46.112289 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:46.112263 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-k6p6q\" (UID: \"6e7dd5b1-9792-4bee-9b5c-f934e2089c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q" Oct 30 13:17:46.114201 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:46.114174 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-k6p6q\" (UID: \"6e7dd5b1-9792-4bee-9b5c-f934e2089c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q" Oct 30 13:17:46.136922 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:46.136851 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxzwz\" (UniqueName: \"kubernetes.io/projected/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-kube-api-access-xxzwz\") pod \"maas-api-revoke-sub-3d4156f1-k6p6q\" (UID: \"6e7dd5b1-9792-4bee-9b5c-f934e2089c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q" Oct 30 13:17:46.199148 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:46.199116 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q" Oct 30 13:17:46.510046 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:46.510011 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q"] Oct 30 13:17:46.511552 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:17:46.511516 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7dd5b1_9792_4bee_9b5c_f934e2089c78.slice/crio-355c81ccf609af25c1f62c8b54d659cc8f4a1e6eeb33ed25187cbbfd133a1964 WatchSource:0}: Error finding container 355c81ccf609af25c1f62c8b54d659cc8f4a1e6eeb33ed25187cbbfd133a1964: Status 404 returned error can't find the container with id 355c81ccf609af25c1f62c8b54d659cc8f4a1e6eeb33ed25187cbbfd133a1964 Oct 30 13:17:46.839387 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:46.839350 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q" event={"ID":"6e7dd5b1-9792-4bee-9b5c-f934e2089c78","Type":"ContainerStarted","Data":"355c81ccf609af25c1f62c8b54d659cc8f4a1e6eeb33ed25187cbbfd133a1964"} Oct 30 13:17:47.846356 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:47.846260 4503 generic.go:356] "Generic (PLEG): container finished" podID="6e7dd5b1-9792-4bee-9b5c-f934e2089c78" containerID="8b6839f2114dacaa9faca439266d107977051e9b05681cadd5481d59f92206e2" exitCode=0 Oct 30 13:17:47.846734 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:47.846344 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q" event={"ID":"6e7dd5b1-9792-4bee-9b5c-f934e2089c78","Type":"ContainerDied","Data":"8b6839f2114dacaa9faca439266d107977051e9b05681cadd5481d59f92206e2"} Oct 30 13:17:48.991696 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:48.991667 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q" Oct 30 13:17:49.041808 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:49.041775 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-maas-api-openshift-ca\") pod \"6e7dd5b1-9792-4bee-9b5c-f934e2089c78\" (UID: \"6e7dd5b1-9792-4bee-9b5c-f934e2089c78\") " Oct 30 13:17:49.042019 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:49.041837 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-kube-api-access-xxzwz\" (UniqueName: \"kubernetes.io/projected/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-kube-api-access-xxzwz\") pod \"6e7dd5b1-9792-4bee-9b5c-f934e2089c78\" (UID: \"6e7dd5b1-9792-4bee-9b5c-f934e2089c78\") " Oct 30 13:17:49.042019 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:49.041874 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-maas-api-xks-ca\") pod \"6e7dd5b1-9792-4bee-9b5c-f934e2089c78\" (UID: \"6e7dd5b1-9792-4bee-9b5c-f934e2089c78\") " Oct 30 13:17:49.042224 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:49.042192 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-maas-api-openshift-ca" pod "6e7dd5b1-9792-4bee-9b5c-f934e2089c78" (UID: "6e7dd5b1-9792-4bee-9b5c-f934e2089c78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:49.042300 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:49.042270 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-maas-api-openshift-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:17:49.044190 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:49.044152 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-maas-api-xks-ca" pod "6e7dd5b1-9792-4bee-9b5c-f934e2089c78" (UID: "6e7dd5b1-9792-4bee-9b5c-f934e2089c78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:17:49.044309 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:49.044203 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-kube-api-access-xxzwz" pod "6e7dd5b1-9792-4bee-9b5c-f934e2089c78" (UID: "6e7dd5b1-9792-4bee-9b5c-f934e2089c78"). InnerVolumeSpecName "kube-api-access-xxzwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:49.143068 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:49.143037 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxzwz\" (UniqueName: \"kubernetes.io/projected/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-kube-api-access-xxzwz\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:17:49.143228 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:49.143073 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e7dd5b1-9792-4bee-9b5c-f934e2089c78-maas-api-xks-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:17:49.860512 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:49.860481 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q" Oct 30 13:17:49.860687 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:49.860510 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q" event={"ID":"6e7dd5b1-9792-4bee-9b5c-f934e2089c78","Type":"ContainerDied","Data":"355c81ccf609af25c1f62c8b54d659cc8f4a1e6eeb33ed25187cbbfd133a1964"} Oct 30 13:17:49.860687 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:49.860555 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="355c81ccf609af25c1f62c8b54d659cc8f4a1e6eeb33ed25187cbbfd133a1964" Oct 30 13:17:49.918505 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:17:49.918470 4503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7dd5b1_9792_4bee_9b5c_f934e2089c78.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:17:56.053481 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:56.053347 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w"] Oct 30 13:17:56.063783 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:56.063746 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-345fe60c-mhg2w"] Oct 30 13:17:56.362788 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:17:56.362712 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08749638-af48-4615-9551-3b34e32233b3" path="/var/lib/kubelet/pods/08749638-af48-4615-9551-3b34e32233b3/volumes" Oct 30 13:17:59.871878 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:17:59.871774 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:18:07.106993 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.106960 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q"] Oct 30 13:18:07.118966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.118924 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k6p6q"] Oct 30 13:18:07.160285 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.160254 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9"] Oct 30 13:18:07.160771 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.160755 4503 cpu_manager.go:414] "containerMap: removing container" podUID="6e7dd5b1-9792-4bee-9b5c-f934e2089c78" containerName="revoke-keys" Oct 30 13:18:07.160815 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.160772 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e7dd5b1-9792-4bee-9b5c-f934e2089c78" containerName="revoke-keys" Oct 30 13:18:07.160883 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.160872 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e7dd5b1-9792-4bee-9b5c-f934e2089c78" containerName="revoke-keys" Oct 30 13:18:07.166531 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.166495 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9" Oct 30 13:18:07.183087 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.183056 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:18:07.183294 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.183181 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:18:07.186687 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.186660 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9"] Oct 30 13:18:07.276388 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.276348 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07afddeb-bc40-4760-ae9c-d9035442e1e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-r69v9\" (UID: \"07afddeb-bc40-4760-ae9c-d9035442e1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9" Oct 30 13:18:07.276388 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.276387 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07afddeb-bc40-4760-ae9c-d9035442e1e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-r69v9\" (UID: \"07afddeb-bc40-4760-ae9c-d9035442e1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9" Oct 30 13:18:07.276607 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.276538 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r2mt\" (UniqueName: \"kubernetes.io/projected/07afddeb-bc40-4760-ae9c-d9035442e1e6-kube-api-access-6r2mt\") pod \"maas-api-revoke-sub-3d4156f1-r69v9\" (UID: \"07afddeb-bc40-4760-ae9c-d9035442e1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9" Oct 30 13:18:07.378047 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.377933 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r2mt\" (UniqueName: \"kubernetes.io/projected/07afddeb-bc40-4760-ae9c-d9035442e1e6-kube-api-access-6r2mt\") pod \"maas-api-revoke-sub-3d4156f1-r69v9\" (UID: \"07afddeb-bc40-4760-ae9c-d9035442e1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9" Oct 30 13:18:07.378192 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.378067 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07afddeb-bc40-4760-ae9c-d9035442e1e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-r69v9\" (UID: \"07afddeb-bc40-4760-ae9c-d9035442e1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9" Oct 30 13:18:07.378192 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.378095 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07afddeb-bc40-4760-ae9c-d9035442e1e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-r69v9\" (UID: \"07afddeb-bc40-4760-ae9c-d9035442e1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9" Oct 30 13:18:07.378913 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.378886 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07afddeb-bc40-4760-ae9c-d9035442e1e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-r69v9\" (UID: \"07afddeb-bc40-4760-ae9c-d9035442e1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9" Oct 30 13:18:07.380479 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.380452 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07afddeb-bc40-4760-ae9c-d9035442e1e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-r69v9\" (UID: \"07afddeb-bc40-4760-ae9c-d9035442e1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9" Oct 30 13:18:07.402353 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.402329 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r2mt\" (UniqueName: \"kubernetes.io/projected/07afddeb-bc40-4760-ae9c-d9035442e1e6-kube-api-access-6r2mt\") pod \"maas-api-revoke-sub-3d4156f1-r69v9\" (UID: \"07afddeb-bc40-4760-ae9c-d9035442e1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9" Oct 30 13:18:07.481140 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.481103 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9" Oct 30 13:18:07.783041 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:07.783010 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9"] Oct 30 13:18:07.783629 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:18:07.783598 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07afddeb_bc40_4760_ae9c_d9035442e1e6.slice/crio-62aad1c5e4310fc20d599c6a8eb03eea562cc584883078f408990a5b75a8a997 WatchSource:0}: Error finding container 62aad1c5e4310fc20d599c6a8eb03eea562cc584883078f408990a5b75a8a997: Status 404 returned error can't find the container with id 62aad1c5e4310fc20d599c6a8eb03eea562cc584883078f408990a5b75a8a997 Oct 30 13:18:08.015889 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:08.015848 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9" event={"ID":"07afddeb-bc40-4760-ae9c-d9035442e1e6","Type":"ContainerStarted","Data":"62aad1c5e4310fc20d599c6a8eb03eea562cc584883078f408990a5b75a8a997"} Oct 30 13:18:08.361706 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:08.361616 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e7dd5b1-9792-4bee-9b5c-f934e2089c78" path="/var/lib/kubelet/pods/6e7dd5b1-9792-4bee-9b5c-f934e2089c78/volumes" Oct 30 13:18:09.022996 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:09.022888 4503 generic.go:356] "Generic (PLEG): container finished" podID="07afddeb-bc40-4760-ae9c-d9035442e1e6" containerID="83539a94cc4887314505dbf06d178fd7ee090e7d5153ba7ec0f1166fc3752040" exitCode=0 Oct 30 13:18:09.022996 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:09.022980 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9" event={"ID":"07afddeb-bc40-4760-ae9c-d9035442e1e6","Type":"ContainerDied","Data":"83539a94cc4887314505dbf06d178fd7ee090e7d5153ba7ec0f1166fc3752040"} Oct 30 13:18:10.171092 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:10.171063 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9" Oct 30 13:18:10.208522 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:10.208480 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07afddeb-bc40-4760-ae9c-d9035442e1e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07afddeb-bc40-4760-ae9c-d9035442e1e6-maas-api-openshift-ca\") pod \"07afddeb-bc40-4760-ae9c-d9035442e1e6\" (UID: \"07afddeb-bc40-4760-ae9c-d9035442e1e6\") " Oct 30 13:18:10.208707 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:10.208593 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07afddeb-bc40-4760-ae9c-d9035442e1e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07afddeb-bc40-4760-ae9c-d9035442e1e6-maas-api-xks-ca\") pod \"07afddeb-bc40-4760-ae9c-d9035442e1e6\" (UID: \"07afddeb-bc40-4760-ae9c-d9035442e1e6\") " Oct 30 13:18:10.208707 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:10.208683 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07afddeb-bc40-4760-ae9c-d9035442e1e6-kube-api-access-6r2mt\" (UniqueName: \"kubernetes.io/projected/07afddeb-bc40-4760-ae9c-d9035442e1e6-kube-api-access-6r2mt\") pod \"07afddeb-bc40-4760-ae9c-d9035442e1e6\" (UID: \"07afddeb-bc40-4760-ae9c-d9035442e1e6\") " Oct 30 13:18:10.208976 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:10.208919 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07afddeb-bc40-4760-ae9c-d9035442e1e6-maas-api-openshift-ca" pod "07afddeb-bc40-4760-ae9c-d9035442e1e6" (UID: "07afddeb-bc40-4760-ae9c-d9035442e1e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:10.209406 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:10.209380 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07afddeb-bc40-4760-ae9c-d9035442e1e6-maas-api-openshift-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:18:10.211380 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:10.211342 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07afddeb-bc40-4760-ae9c-d9035442e1e6-maas-api-xks-ca" pod "07afddeb-bc40-4760-ae9c-d9035442e1e6" (UID: "07afddeb-bc40-4760-ae9c-d9035442e1e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:18:10.211502 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:10.211467 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07afddeb-bc40-4760-ae9c-d9035442e1e6-kube-api-access-6r2mt" pod "07afddeb-bc40-4760-ae9c-d9035442e1e6" (UID: "07afddeb-bc40-4760-ae9c-d9035442e1e6"). InnerVolumeSpecName "kube-api-access-6r2mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:10.309957 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:10.309847 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r2mt\" (UniqueName: \"kubernetes.io/projected/07afddeb-bc40-4760-ae9c-d9035442e1e6-kube-api-access-6r2mt\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:18:10.309957 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:10.309892 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07afddeb-bc40-4760-ae9c-d9035442e1e6-maas-api-xks-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:18:10.408927 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:18:10.408889 4503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07afddeb_bc40_4760_ae9c_d9035442e1e6.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:18:11.036784 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:11.036754 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9" Oct 30 13:18:11.037006 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:11.036798 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9" event={"ID":"07afddeb-bc40-4760-ae9c-d9035442e1e6","Type":"ContainerDied","Data":"62aad1c5e4310fc20d599c6a8eb03eea562cc584883078f408990a5b75a8a997"} Oct 30 13:18:11.037006 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:11.036868 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62aad1c5e4310fc20d599c6a8eb03eea562cc584883078f408990a5b75a8a997" Oct 30 13:18:26.046317 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:26.046269 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7"] Oct 30 13:18:26.059723 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:26.059687 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88b26747-r5rm7"] Oct 30 13:18:26.363085 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:26.362990 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8d6f01a-f566-41ec-8d49-5dbc377ea951" path="/var/lib/kubelet/pods/c8d6f01a-f566-41ec-8d49-5dbc377ea951/volumes" Oct 30 13:18:29.700525 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:18:29.700487 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:18:30.769644 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:30.769601 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9"] Oct 30 13:18:30.781876 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:30.781846 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r69v9"] Oct 30 13:18:32.360763 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:32.360730 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07afddeb-bc40-4760-ae9c-d9035442e1e6" path="/var/lib/kubelet/pods/07afddeb-bc40-4760-ae9c-d9035442e1e6/volumes" Oct 30 13:18:47.780475 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:47.780430 4503 scope.go:122] "RemoveContainer" containerID="5ccf727a9eab881fef4dd8c31f67fd24a70f68d885a1ba8c25dcdb63cad186a1" Oct 30 13:18:58.549598 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.549561 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq"] Oct 30 13:18:58.550183 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.550165 4503 cpu_manager.go:414] "containerMap: removing container" podUID="07afddeb-bc40-4760-ae9c-d9035442e1e6" containerName="revoke-keys" Oct 30 13:18:58.550239 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.550184 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07afddeb-bc40-4760-ae9c-d9035442e1e6" containerName="revoke-keys" Oct 30 13:18:58.550307 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.550294 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="07afddeb-bc40-4760-ae9c-d9035442e1e6" containerName="revoke-keys" Oct 30 13:18:58.555221 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.555185 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" Oct 30 13:18:58.583916 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.583882 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-cadb6b-serving-cert\"" Oct 30 13:18:58.584130 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.584078 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d5n24\"" Oct 30 13:18:58.584130 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.584084 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 13:18:58.587128 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.587098 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq"] Oct 30 13:18:58.609989 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.609919 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c02b0526-889e-4e0a-b9a1-a948015fde4b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq\" (UID: \"c02b0526-889e-4e0a-b9a1-a948015fde4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" Oct 30 13:18:58.610171 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.610097 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c02b0526-889e-4e0a-b9a1-a948015fde4b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq\" (UID: \"c02b0526-889e-4e0a-b9a1-a948015fde4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" Oct 30 13:18:58.610171 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.610135 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02b0526-889e-4e0a-b9a1-a948015fde4b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq\" (UID: \"c02b0526-889e-4e0a-b9a1-a948015fde4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" Oct 30 13:18:58.610171 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.610161 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hj64\" (UniqueName: \"kubernetes.io/projected/c02b0526-889e-4e0a-b9a1-a948015fde4b-kube-api-access-6hj64\") pod \"maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq\" (UID: \"c02b0526-889e-4e0a-b9a1-a948015fde4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" Oct 30 13:18:58.710844 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.710800 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c02b0526-889e-4e0a-b9a1-a948015fde4b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq\" (UID: \"c02b0526-889e-4e0a-b9a1-a948015fde4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" Oct 30 13:18:58.711108 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.710913 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c02b0526-889e-4e0a-b9a1-a948015fde4b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq\" (UID: \"c02b0526-889e-4e0a-b9a1-a948015fde4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" Oct 30 13:18:58.711108 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:18:58.710991 4503 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-serving-cert: secret "maas-api-e2e-worker-single-main-cadb6b-serving-cert" not found Oct 30 13:18:58.711108 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:18:58.711066 4503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02b0526-889e-4e0a-b9a1-a948015fde4b-maas-api-tls podName:c02b0526-889e-4e0a-b9a1-a948015fde4b nodeName:}" failed. No retries permitted until 2026-10-30 13:18:59.2110499 +0000 UTC m=+1633.034278160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c02b0526-889e-4e0a-b9a1-a948015fde4b-maas-api-tls") pod "maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" (UID: "c02b0526-889e-4e0a-b9a1-a948015fde4b") : secret "maas-api-e2e-worker-single-main-cadb6b-serving-cert" not found Oct 30 13:18:58.711108 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.711082 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02b0526-889e-4e0a-b9a1-a948015fde4b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq\" (UID: \"c02b0526-889e-4e0a-b9a1-a948015fde4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" Oct 30 13:18:58.711108 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.711106 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hj64\" (UniqueName: \"kubernetes.io/projected/c02b0526-889e-4e0a-b9a1-a948015fde4b-kube-api-access-6hj64\") pod \"maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq\" (UID: \"c02b0526-889e-4e0a-b9a1-a948015fde4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" Oct 30 13:18:58.711811 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.711779 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c02b0526-889e-4e0a-b9a1-a948015fde4b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq\" (UID: \"c02b0526-889e-4e0a-b9a1-a948015fde4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" Oct 30 13:18:58.714002 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.713966 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02b0526-889e-4e0a-b9a1-a948015fde4b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq\" (UID: \"c02b0526-889e-4e0a-b9a1-a948015fde4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" Oct 30 13:18:58.738167 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:58.738123 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hj64\" (UniqueName: \"kubernetes.io/projected/c02b0526-889e-4e0a-b9a1-a948015fde4b-kube-api-access-6hj64\") pod \"maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq\" (UID: \"c02b0526-889e-4e0a-b9a1-a948015fde4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" Oct 30 13:18:59.216339 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:59.216298 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c02b0526-889e-4e0a-b9a1-a948015fde4b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq\" (UID: \"c02b0526-889e-4e0a-b9a1-a948015fde4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" Oct 30 13:18:59.218785 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:59.218759 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c02b0526-889e-4e0a-b9a1-a948015fde4b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq\" (UID: \"c02b0526-889e-4e0a-b9a1-a948015fde4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" Oct 30 13:18:59.470246 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:59.470161 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" Oct 30 13:18:59.700920 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:18:59.700880 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:18:59.793794 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:18:59.792597 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq"] Oct 30 13:18:59.793920 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:18:59.793889 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02b0526_889e_4e0a_b9a1_a948015fde4b.slice/crio-21506722e90f8868994b73a27815fc6e9c483f8a28950ee8274070a4d1793f7d WatchSource:0}: Error finding container 21506722e90f8868994b73a27815fc6e9c483f8a28950ee8274070a4d1793f7d: Status 404 returned error can't find the container with id 21506722e90f8868994b73a27815fc6e9c483f8a28950ee8274070a4d1793f7d Oct 30 13:19:00.337539 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:19:00.337502 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" event={"ID":"c02b0526-889e-4e0a-b9a1-a948015fde4b","Type":"ContainerStarted","Data":"07fececb5de3ded82289263abc4e44e6638d6aa0f6446992f5254dde2ab6427c"} Oct 30 13:19:00.337755 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:19:00.337577 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" event={"ID":"c02b0526-889e-4e0a-b9a1-a948015fde4b","Type":"ContainerStarted","Data":"21506722e90f8868994b73a27815fc6e9c483f8a28950ee8274070a4d1793f7d"} Oct 30 13:19:00.337755 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:19:00.337709 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" Oct 30 13:19:00.384855 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:19:00.384801 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" podStartSLOduration=2.011498595 podStartE2EDuration="2.384782505s" totalImagesPullingTime="373.28391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:00.383993984 +0000 UTC m=+1634.207222261" watchObservedRunningTime="2026-10-30 13:19:00.384782505 +0000 UTC m=+1634.208010781" Oct 30 13:19:06.351001 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:19:06.350968 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" Oct 30 13:19:15.036203 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:19:15.036161 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm"] Oct 30 13:19:15.047349 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:19:15.047319 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t7sjm"] Oct 30 13:19:16.370107 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:19:16.370072 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2b4f45d-b484-48d6-b6a6-43739a937815" path="/var/lib/kubelet/pods/a2b4f45d-b484-48d6-b6a6-43739a937815/volumes" Oct 30 13:19:29.701340 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:19:29.701258 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:19:47.807151 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:19:47.807115 4503 scope.go:122] "RemoveContainer" containerID="3a6a88cb2c061356dd5cee21d712e8e9ae965a862e33911621af3d7d4c005447" Oct 30 13:19:47.816404 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:19:47.816379 4503 scope.go:122] "RemoveContainer" containerID="1713a29c32a56ed21700c7c63291197a245fe83fa939a88af16b35c093efac05" Oct 30 13:19:59.700651 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:19:59.700612 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:20:02.538794 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:02.538758 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq"] Oct 30 13:20:02.539349 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:02.539028 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" podUID="c02b0526-889e-4e0a-b9a1-a948015fde4b" containerName="maas-api" containerID="cri-o://07fececb5de3ded82289263abc4e44e6638d6aa0f6446992f5254dde2ab6427c" gracePeriod=30 Oct 30 13:20:03.034193 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.034165 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" Oct 30 13:20:03.099702 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.099601 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c02b0526-889e-4e0a-b9a1-a948015fde4b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02b0526-889e-4e0a-b9a1-a948015fde4b-metrics-certs\") pod \"c02b0526-889e-4e0a-b9a1-a948015fde4b\" (UID: \"c02b0526-889e-4e0a-b9a1-a948015fde4b\") " Oct 30 13:20:03.099702 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.099650 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c02b0526-889e-4e0a-b9a1-a948015fde4b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c02b0526-889e-4e0a-b9a1-a948015fde4b-maas-api-tls\") pod \"c02b0526-889e-4e0a-b9a1-a948015fde4b\" (UID: \"c02b0526-889e-4e0a-b9a1-a948015fde4b\") " Oct 30 13:20:03.099982 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.099782 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c02b0526-889e-4e0a-b9a1-a948015fde4b-kube-api-access-6hj64\" (UniqueName: \"kubernetes.io/projected/c02b0526-889e-4e0a-b9a1-a948015fde4b-kube-api-access-6hj64\") pod \"c02b0526-889e-4e0a-b9a1-a948015fde4b\" (UID: \"c02b0526-889e-4e0a-b9a1-a948015fde4b\") " Oct 30 13:20:03.099982 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.099822 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c02b0526-889e-4e0a-b9a1-a948015fde4b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c02b0526-889e-4e0a-b9a1-a948015fde4b-openshift-service-ca\") pod \"c02b0526-889e-4e0a-b9a1-a948015fde4b\" (UID: \"c02b0526-889e-4e0a-b9a1-a948015fde4b\") " Oct 30 13:20:03.100307 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.100275 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c02b0526-889e-4e0a-b9a1-a948015fde4b-openshift-service-ca" pod "c02b0526-889e-4e0a-b9a1-a948015fde4b" (UID: "c02b0526-889e-4e0a-b9a1-a948015fde4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:20:03.102086 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.102061 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c02b0526-889e-4e0a-b9a1-a948015fde4b-metrics-certs" pod "c02b0526-889e-4e0a-b9a1-a948015fde4b" (UID: "c02b0526-889e-4e0a-b9a1-a948015fde4b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:20:03.102215 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.102092 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c02b0526-889e-4e0a-b9a1-a948015fde4b-maas-api-tls" pod "c02b0526-889e-4e0a-b9a1-a948015fde4b" (UID: "c02b0526-889e-4e0a-b9a1-a948015fde4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:20:03.102215 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.102095 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02b0526-889e-4e0a-b9a1-a948015fde4b-kube-api-access-6hj64" pod "c02b0526-889e-4e0a-b9a1-a948015fde4b" (UID: "c02b0526-889e-4e0a-b9a1-a948015fde4b"). InnerVolumeSpecName "kube-api-access-6hj64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:03.201158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.201128 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hj64\" (UniqueName: \"kubernetes.io/projected/c02b0526-889e-4e0a-b9a1-a948015fde4b-kube-api-access-6hj64\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:20:03.201158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.201155 4503 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c02b0526-889e-4e0a-b9a1-a948015fde4b-openshift-service-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:20:03.201158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.201164 4503 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02b0526-889e-4e0a-b9a1-a948015fde4b-metrics-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:20:03.201408 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.201173 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c02b0526-889e-4e0a-b9a1-a948015fde4b-maas-api-tls\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:20:03.745713 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.745670 4503 generic.go:356] "Generic (PLEG): container finished" podID="c02b0526-889e-4e0a-b9a1-a948015fde4b" containerID="07fececb5de3ded82289263abc4e44e6638d6aa0f6446992f5254dde2ab6427c" exitCode=0 Oct 30 13:20:03.746262 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.745725 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" Oct 30 13:20:03.746262 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.745732 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" event={"ID":"c02b0526-889e-4e0a-b9a1-a948015fde4b","Type":"ContainerDied","Data":"07fececb5de3ded82289263abc4e44e6638d6aa0f6446992f5254dde2ab6427c"} Oct 30 13:20:03.746262 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.745848 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq" event={"ID":"c02b0526-889e-4e0a-b9a1-a948015fde4b","Type":"ContainerDied","Data":"21506722e90f8868994b73a27815fc6e9c483f8a28950ee8274070a4d1793f7d"} Oct 30 13:20:03.746262 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.745870 4503 scope.go:122] "RemoveContainer" containerID="07fececb5de3ded82289263abc4e44e6638d6aa0f6446992f5254dde2ab6427c" Oct 30 13:20:03.757154 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.757107 4503 scope.go:122] "RemoveContainer" containerID="07fececb5de3ded82289263abc4e44e6638d6aa0f6446992f5254dde2ab6427c" Oct 30 13:20:03.757476 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:20:03.757453 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07fececb5de3ded82289263abc4e44e6638d6aa0f6446992f5254dde2ab6427c\": container with ID starting with 07fececb5de3ded82289263abc4e44e6638d6aa0f6446992f5254dde2ab6427c not found: ID does not exist" containerID="07fececb5de3ded82289263abc4e44e6638d6aa0f6446992f5254dde2ab6427c" Oct 30 13:20:03.757533 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.757487 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fececb5de3ded82289263abc4e44e6638d6aa0f6446992f5254dde2ab6427c"} err="failed to get container status \"07fececb5de3ded82289263abc4e44e6638d6aa0f6446992f5254dde2ab6427c\": rpc error: code = NotFound desc = could not find container \"07fececb5de3ded82289263abc4e44e6638d6aa0f6446992f5254dde2ab6427c\": container with ID starting with 07fececb5de3ded82289263abc4e44e6638d6aa0f6446992f5254dde2ab6427c not found: ID does not exist" Oct 30 13:20:03.800895 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.800858 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq"] Oct 30 13:20:03.813129 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:03.813094 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cadb6b-b56cbfcc5-gbfbq"] Oct 30 13:20:04.361136 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:04.361102 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c02b0526-889e-4e0a-b9a1-a948015fde4b" path="/var/lib/kubelet/pods/c02b0526-889e-4e0a-b9a1-a948015fde4b/volumes" Oct 30 13:20:12.337801 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:12.337767 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q"] Oct 30 13:20:12.338367 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:12.338348 4503 cpu_manager.go:414] "containerMap: removing container" podUID="c02b0526-889e-4e0a-b9a1-a948015fde4b" containerName="maas-api" Oct 30 13:20:12.338412 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:12.338369 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c02b0526-889e-4e0a-b9a1-a948015fde4b" containerName="maas-api" Oct 30 13:20:12.338477 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:12.338466 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="c02b0526-889e-4e0a-b9a1-a948015fde4b" containerName="maas-api" Oct 30 13:20:12.341628 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:12.341611 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q" Oct 30 13:20:12.364507 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:12.364477 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q"] Oct 30 13:20:12.375068 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:12.375034 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:20:12.375224 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:12.375202 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:20:12.392498 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:12.392446 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khgxk\" (UniqueName: \"kubernetes.io/projected/8b846c45-29d4-4995-b2a5-98fdc1f20954-kube-api-access-khgxk\") pod \"maas-api-revoke-sub-3d4156f1-x858q\" (UID: \"8b846c45-29d4-4995-b2a5-98fdc1f20954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q" Oct 30 13:20:12.392674 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:12.392595 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b846c45-29d4-4995-b2a5-98fdc1f20954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-x858q\" (UID: \"8b846c45-29d4-4995-b2a5-98fdc1f20954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q" Oct 30 13:20:12.392757 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:12.392740 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b846c45-29d4-4995-b2a5-98fdc1f20954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-x858q\" (UID: \"8b846c45-29d4-4995-b2a5-98fdc1f20954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q" Oct 30 13:20:12.493950 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:12.493904 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b846c45-29d4-4995-b2a5-98fdc1f20954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-x858q\" (UID: \"8b846c45-29d4-4995-b2a5-98fdc1f20954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q" Oct 30 13:20:12.494147 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:12.494026 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khgxk\" (UniqueName: \"kubernetes.io/projected/8b846c45-29d4-4995-b2a5-98fdc1f20954-kube-api-access-khgxk\") pod \"maas-api-revoke-sub-3d4156f1-x858q\" (UID: \"8b846c45-29d4-4995-b2a5-98fdc1f20954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q" Oct 30 13:20:12.494147 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:12.494069 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b846c45-29d4-4995-b2a5-98fdc1f20954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-x858q\" (UID: \"8b846c45-29d4-4995-b2a5-98fdc1f20954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q" Oct 30 13:20:12.494598 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:12.494575 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b846c45-29d4-4995-b2a5-98fdc1f20954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-x858q\" (UID: \"8b846c45-29d4-4995-b2a5-98fdc1f20954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q" Oct 30 13:20:12.496487 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:12.496468 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b846c45-29d4-4995-b2a5-98fdc1f20954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-x858q\" (UID: \"8b846c45-29d4-4995-b2a5-98fdc1f20954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q" Oct 30 13:20:12.519199 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:12.519169 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khgxk\" (UniqueName: \"kubernetes.io/projected/8b846c45-29d4-4995-b2a5-98fdc1f20954-kube-api-access-khgxk\") pod \"maas-api-revoke-sub-3d4156f1-x858q\" (UID: \"8b846c45-29d4-4995-b2a5-98fdc1f20954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q" Oct 30 13:20:12.654330 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:12.654289 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q" Oct 30 13:20:12.956293 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:12.956262 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q"] Oct 30 13:20:12.956914 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:20:12.956884 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b846c45_29d4_4995_b2a5_98fdc1f20954.slice/crio-57179e8e32dd3eed97f96dc80930b728d75314af6271c4fc8856bf829134e02c WatchSource:0}: Error finding container 57179e8e32dd3eed97f96dc80930b728d75314af6271c4fc8856bf829134e02c: Status 404 returned error can't find the container with id 57179e8e32dd3eed97f96dc80930b728d75314af6271c4fc8856bf829134e02c Oct 30 13:20:13.826886 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:13.826848 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q" event={"ID":"8b846c45-29d4-4995-b2a5-98fdc1f20954","Type":"ContainerStarted","Data":"34cbd05f4deb8fed8fcf3929ba5bcebcbd0c41b10735de1d61179f3bc629a0db"} Oct 30 13:20:13.827317 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:13.826903 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q" event={"ID":"8b846c45-29d4-4995-b2a5-98fdc1f20954","Type":"ContainerStarted","Data":"57179e8e32dd3eed97f96dc80930b728d75314af6271c4fc8856bf829134e02c"} Oct 30 13:20:13.871074 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:13.871025 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q" podStartSLOduration=1.8710125199999998 podStartE2EDuration="1.87101252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:13.870683163 +0000 UTC m=+1707.693911440" watchObservedRunningTime="2026-10-30 13:20:13.87101252 +0000 UTC m=+1707.694240831" Oct 30 13:20:14.833878 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:14.833846 4503 generic.go:356] "Generic (PLEG): container finished" podID="8b846c45-29d4-4995-b2a5-98fdc1f20954" containerID="34cbd05f4deb8fed8fcf3929ba5bcebcbd0c41b10735de1d61179f3bc629a0db" exitCode=0 Oct 30 13:20:14.834264 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:14.833958 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q" event={"ID":"8b846c45-29d4-4995-b2a5-98fdc1f20954","Type":"ContainerDied","Data":"34cbd05f4deb8fed8fcf3929ba5bcebcbd0c41b10735de1d61179f3bc629a0db"} Oct 30 13:20:15.978462 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:15.978431 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q" Oct 30 13:20:16.030002 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:16.029911 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b846c45-29d4-4995-b2a5-98fdc1f20954-kube-api-access-khgxk\" (UniqueName: \"kubernetes.io/projected/8b846c45-29d4-4995-b2a5-98fdc1f20954-kube-api-access-khgxk\") pod \"8b846c45-29d4-4995-b2a5-98fdc1f20954\" (UID: \"8b846c45-29d4-4995-b2a5-98fdc1f20954\") " Oct 30 13:20:16.030183 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:16.030088 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b846c45-29d4-4995-b2a5-98fdc1f20954-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b846c45-29d4-4995-b2a5-98fdc1f20954-maas-api-xks-ca\") pod \"8b846c45-29d4-4995-b2a5-98fdc1f20954\" (UID: \"8b846c45-29d4-4995-b2a5-98fdc1f20954\") " Oct 30 13:20:16.030259 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:16.030225 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b846c45-29d4-4995-b2a5-98fdc1f20954-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b846c45-29d4-4995-b2a5-98fdc1f20954-maas-api-openshift-ca\") pod \"8b846c45-29d4-4995-b2a5-98fdc1f20954\" (UID: \"8b846c45-29d4-4995-b2a5-98fdc1f20954\") " Oct 30 13:20:16.030569 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:16.030544 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b846c45-29d4-4995-b2a5-98fdc1f20954-maas-api-openshift-ca" pod "8b846c45-29d4-4995-b2a5-98fdc1f20954" (UID: "8b846c45-29d4-4995-b2a5-98fdc1f20954"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:20:16.032586 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:16.032559 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b846c45-29d4-4995-b2a5-98fdc1f20954-maas-api-xks-ca" pod "8b846c45-29d4-4995-b2a5-98fdc1f20954" (UID: "8b846c45-29d4-4995-b2a5-98fdc1f20954"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:20:16.032586 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:16.032568 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b846c45-29d4-4995-b2a5-98fdc1f20954-kube-api-access-khgxk" pod "8b846c45-29d4-4995-b2a5-98fdc1f20954" (UID: "8b846c45-29d4-4995-b2a5-98fdc1f20954"). InnerVolumeSpecName "kube-api-access-khgxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:16.131801 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:16.131718 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b846c45-29d4-4995-b2a5-98fdc1f20954-maas-api-openshift-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:20:16.131801 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:16.131747 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khgxk\" (UniqueName: \"kubernetes.io/projected/8b846c45-29d4-4995-b2a5-98fdc1f20954-kube-api-access-khgxk\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:20:16.131801 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:16.131758 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b846c45-29d4-4995-b2a5-98fdc1f20954-maas-api-xks-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:20:16.847152 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:16.847114 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q" event={"ID":"8b846c45-29d4-4995-b2a5-98fdc1f20954","Type":"ContainerDied","Data":"57179e8e32dd3eed97f96dc80930b728d75314af6271c4fc8856bf829134e02c"} Oct 30 13:20:16.847333 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:16.847162 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q" Oct 30 13:20:16.847333 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:16.847164 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57179e8e32dd3eed97f96dc80930b728d75314af6271c4fc8856bf829134e02c" Oct 30 13:20:28.036770 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:28.036735 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc"] Oct 30 13:20:28.048362 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:28.048336 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2z7cc"] Oct 30 13:20:28.361928 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:28.361849 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c015946-f285-4a73-ab87-d747da2b3509" path="/var/lib/kubelet/pods/6c015946-f285-4a73-ab87-d747da2b3509/volumes" Oct 30 13:20:29.700505 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:20:29.700467 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:20:36.441489 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:36.441452 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt"] Oct 30 13:20:36.441932 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:36.441652 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt" podUID="2ecba625-9f77-49d0-9f9f-056fc89a6271" containerName="manager" containerID="cri-o://f63eafd348c7a23caf5a9da5806c7b7834167f56f08d80174afd1d01eca9fd0d" gracePeriod=10 Oct 30 13:20:36.707301 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:36.707236 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt" Oct 30 13:20:36.725894 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:36.725859 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecba625-9f77-49d0-9f9f-056fc89a6271-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ecba625-9f77-49d0-9f9f-056fc89a6271-extensions-socket-volume\") pod \"2ecba625-9f77-49d0-9f9f-056fc89a6271\" (UID: \"2ecba625-9f77-49d0-9f9f-056fc89a6271\") " Oct 30 13:20:36.726059 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:36.725904 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ecba625-9f77-49d0-9f9f-056fc89a6271-kube-api-access-hf4mm\" (UniqueName: \"kubernetes.io/projected/2ecba625-9f77-49d0-9f9f-056fc89a6271-kube-api-access-hf4mm\") pod \"2ecba625-9f77-49d0-9f9f-056fc89a6271\" (UID: \"2ecba625-9f77-49d0-9f9f-056fc89a6271\") " Oct 30 13:20:36.726246 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:36.726221 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecba625-9f77-49d0-9f9f-056fc89a6271-extensions-socket-volume" pod "2ecba625-9f77-49d0-9f9f-056fc89a6271" (UID: "2ecba625-9f77-49d0-9f9f-056fc89a6271"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:36.727960 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:36.727912 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecba625-9f77-49d0-9f9f-056fc89a6271-kube-api-access-hf4mm" pod "2ecba625-9f77-49d0-9f9f-056fc89a6271" (UID: "2ecba625-9f77-49d0-9f9f-056fc89a6271"). InnerVolumeSpecName "kube-api-access-hf4mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:36.827174 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:36.827143 4503 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ecba625-9f77-49d0-9f9f-056fc89a6271-extensions-socket-volume\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:20:36.827174 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:36.827170 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf4mm\" (UniqueName: \"kubernetes.io/projected/2ecba625-9f77-49d0-9f9f-056fc89a6271-kube-api-access-hf4mm\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:20:37.070220 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:37.070134 4503 generic.go:356] "Generic (PLEG): container finished" podID="2ecba625-9f77-49d0-9f9f-056fc89a6271" containerID="f63eafd348c7a23caf5a9da5806c7b7834167f56f08d80174afd1d01eca9fd0d" exitCode=0 Oct 30 13:20:37.070220 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:37.070183 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt" Oct 30 13:20:37.070398 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:37.070220 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt" event={"ID":"2ecba625-9f77-49d0-9f9f-056fc89a6271","Type":"ContainerDied","Data":"f63eafd348c7a23caf5a9da5806c7b7834167f56f08d80174afd1d01eca9fd0d"} Oct 30 13:20:37.070398 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:37.070277 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt" event={"ID":"2ecba625-9f77-49d0-9f9f-056fc89a6271","Type":"ContainerDied","Data":"4a51fa1dc1c43a863b8578aeeef51822297d4aa4c5ae9f51725e473b250bee69"} Oct 30 13:20:37.070398 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:37.070301 4503 scope.go:122] "RemoveContainer" containerID="f63eafd348c7a23caf5a9da5806c7b7834167f56f08d80174afd1d01eca9fd0d" Oct 30 13:20:37.081696 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:37.081678 4503 scope.go:122] "RemoveContainer" containerID="f63eafd348c7a23caf5a9da5806c7b7834167f56f08d80174afd1d01eca9fd0d" Oct 30 13:20:37.081964 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:20:37.081927 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63eafd348c7a23caf5a9da5806c7b7834167f56f08d80174afd1d01eca9fd0d\": container with ID starting with f63eafd348c7a23caf5a9da5806c7b7834167f56f08d80174afd1d01eca9fd0d not found: ID does not exist" containerID="f63eafd348c7a23caf5a9da5806c7b7834167f56f08d80174afd1d01eca9fd0d" Oct 30 13:20:37.082044 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:37.081977 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63eafd348c7a23caf5a9da5806c7b7834167f56f08d80174afd1d01eca9fd0d"} err="failed to get container status \"f63eafd348c7a23caf5a9da5806c7b7834167f56f08d80174afd1d01eca9fd0d\": rpc error: code = NotFound desc = could not find container \"f63eafd348c7a23caf5a9da5806c7b7834167f56f08d80174afd1d01eca9fd0d\": container with ID starting with f63eafd348c7a23caf5a9da5806c7b7834167f56f08d80174afd1d01eca9fd0d not found: ID does not exist" Oct 30 13:20:37.125714 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:37.125689 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt"] Oct 30 13:20:37.137875 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:37.137843 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4g8bt"] Oct 30 13:20:38.360687 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:38.360653 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ecba625-9f77-49d0-9f9f-056fc89a6271" path="/var/lib/kubelet/pods/2ecba625-9f77-49d0-9f9f-056fc89a6271/volumes" Oct 30 13:20:40.033845 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:40.033808 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6"] Oct 30 13:20:40.045606 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:40.045571 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5fct6"] Oct 30 13:20:40.360993 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:40.360957 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56174a2e-3289-430d-9f76-0161d89bfcde" path="/var/lib/kubelet/pods/56174a2e-3289-430d-9f76-0161d89bfcde/volumes" Oct 30 13:20:47.843614 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:20:47.843573 4503 scope.go:122] "RemoveContainer" containerID="cb0580a66bfe10ef2aa1d3e7ee968047c5979e562a4d462213aee023fad47b9a" Oct 30 13:20:59.701090 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:20:59.701007 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:21:08.035021 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:08.034985 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc"] Oct 30 13:21:08.046872 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:08.046841 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6svcc"] Oct 30 13:21:08.361103 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:08.361021 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bb256e4-67a7-4959-86fa-632133eeeb4e" path="/var/lib/kubelet/pods/0bb256e4-67a7-4959-86fa-632133eeeb4e/volumes" Oct 30 13:21:19.032547 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:19.032508 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8"] Oct 30 13:21:19.044658 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:19.044629 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8s9p8"] Oct 30 13:21:20.360519 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:20.360488 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d49097c-84e1-46df-9d5a-a855d38c3831" path="/var/lib/kubelet/pods/6d49097c-84e1-46df-9d5a-a855d38c3831/volumes" Oct 30 13:21:29.700875 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:21:29.700837 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:21:39.587597 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:39.587559 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-9s5j4"] Oct 30 13:21:39.590145 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:39.588169 4503 cpu_manager.go:414] "containerMap: removing container" podUID="2ecba625-9f77-49d0-9f9f-056fc89a6271" containerName="manager" Oct 30 13:21:39.590145 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:39.588188 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ecba625-9f77-49d0-9f9f-056fc89a6271" containerName="manager" Oct 30 13:21:39.590145 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:39.588215 4503 cpu_manager.go:414] "containerMap: removing container" podUID="8b846c45-29d4-4995-b2a5-98fdc1f20954" containerName="revoke-keys" Oct 30 13:21:39.590145 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:39.588221 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b846c45-29d4-4995-b2a5-98fdc1f20954" containerName="revoke-keys" Oct 30 13:21:39.590145 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:39.588294 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ecba625-9f77-49d0-9f9f-056fc89a6271" containerName="manager" Oct 30 13:21:39.590145 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:39.588305 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b846c45-29d4-4995-b2a5-98fdc1f20954" containerName="revoke-keys" Oct 30 13:21:39.591341 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:39.591322 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-9s5j4" Oct 30 13:21:39.606765 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:39.606728 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-9s5j4"] Oct 30 13:21:39.615819 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:39.615794 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-t2snd\"" Oct 30 13:21:39.735185 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:39.735151 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae9ceeec-34bb-44cf-8319-c32da817aae4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-9s5j4\" (UID: \"ae9ceeec-34bb-44cf-8319-c32da817aae4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-9s5j4" Oct 30 13:21:39.735394 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:39.735204 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gksxs\" (UniqueName: \"kubernetes.io/projected/ae9ceeec-34bb-44cf-8319-c32da817aae4-kube-api-access-gksxs\") pod \"kuadrant-operator-controller-manager-57bb9bfff-9s5j4\" (UID: \"ae9ceeec-34bb-44cf-8319-c32da817aae4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-9s5j4" Oct 30 13:21:39.836538 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:39.836499 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae9ceeec-34bb-44cf-8319-c32da817aae4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-9s5j4\" (UID: \"ae9ceeec-34bb-44cf-8319-c32da817aae4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-9s5j4" Oct 30 13:21:39.836538 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:39.836542 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gksxs\" (UniqueName: \"kubernetes.io/projected/ae9ceeec-34bb-44cf-8319-c32da817aae4-kube-api-access-gksxs\") pod \"kuadrant-operator-controller-manager-57bb9bfff-9s5j4\" (UID: \"ae9ceeec-34bb-44cf-8319-c32da817aae4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-9s5j4" Oct 30 13:21:39.836968 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:39.836912 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae9ceeec-34bb-44cf-8319-c32da817aae4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-9s5j4\" (UID: \"ae9ceeec-34bb-44cf-8319-c32da817aae4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-9s5j4" Oct 30 13:21:39.866687 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:39.866621 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gksxs\" (UniqueName: \"kubernetes.io/projected/ae9ceeec-34bb-44cf-8319-c32da817aae4-kube-api-access-gksxs\") pod \"kuadrant-operator-controller-manager-57bb9bfff-9s5j4\" (UID: \"ae9ceeec-34bb-44cf-8319-c32da817aae4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-9s5j4" Oct 30 13:21:39.904661 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:39.904631 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-9s5j4" Oct 30 13:21:40.230033 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:40.229999 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-9s5j4"] Oct 30 13:21:40.241604 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:21:40.241575 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9ceeec_34bb_44cf_8319_c32da817aae4.slice/crio-4b32a237ba68ff9a03ba01b98031e2c78e52ef4f1d09751231bd0cc1b1ae5760 WatchSource:0}: Error finding container 4b32a237ba68ff9a03ba01b98031e2c78e52ef4f1d09751231bd0cc1b1ae5760: Status 404 returned error can't find the container with id 4b32a237ba68ff9a03ba01b98031e2c78e52ef4f1d09751231bd0cc1b1ae5760 Oct 30 13:21:40.433227 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:40.433182 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-9s5j4" event={"ID":"ae9ceeec-34bb-44cf-8319-c32da817aae4","Type":"ContainerStarted","Data":"95f8b047dea0350ced8cb04a6408431e8076f8516f537aa7d607515a52792556"} Oct 30 13:21:40.433411 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:40.433254 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-9s5j4" event={"ID":"ae9ceeec-34bb-44cf-8319-c32da817aae4","Type":"ContainerStarted","Data":"4b32a237ba68ff9a03ba01b98031e2c78e52ef4f1d09751231bd0cc1b1ae5760"} Oct 30 13:21:40.433411 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:40.433375 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-9s5j4" Oct 30 13:21:40.482743 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:40.482641 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-9s5j4" podStartSLOduration=1.482625925 podStartE2EDuration="1.482625925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:40.480577193 +0000 UTC m=+1794.303805469" watchObservedRunningTime="2026-10-30 13:21:40.482625925 +0000 UTC m=+1794.305854202" Oct 30 13:21:41.041547 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:41.041506 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr"] Oct 30 13:21:41.048485 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:41.048456 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7phhr"] Oct 30 13:21:42.361100 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:42.361068 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5388e3b6-9852-4c65-a53f-eb03413fe27d" path="/var/lib/kubelet/pods/5388e3b6-9852-4c65-a53f-eb03413fe27d/volumes" Oct 30 13:21:47.873851 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:47.873815 4503 scope.go:122] "RemoveContainer" containerID="9eed11350005b5e29dbf79272bdbdc1e9cc05176d353976488c24a2c77ca2a71" Oct 30 13:21:47.882744 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:47.882723 4503 scope.go:122] "RemoveContainer" containerID="35b69ac95b5d6644eadafb5ee7cae4b84789cd839b7a4524869d34db7fda9d6f" Oct 30 13:21:51.441204 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:21:51.441170 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-9s5j4" Oct 30 13:21:59.805225 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:21:59.805179 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:22:01.936864 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:01.936803 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh"] Oct 30 13:22:01.941920 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:01.941899 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh" Oct 30 13:22:01.962554 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:01.962526 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh"] Oct 30 13:22:01.970830 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:01.970807 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:22:01.970982 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:01.970962 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:22:02.035123 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:02.035086 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6"] Oct 30 13:22:02.041706 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:02.041666 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbq6"] Oct 30 13:22:02.057550 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:02.057515 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-5r8dh\" (UID: \"3cb49bf1-1cc0-4b74-b3cb-0e326ce064df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh" Oct 30 13:22:02.057738 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:02.057701 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-5r8dh\" (UID: \"3cb49bf1-1cc0-4b74-b3cb-0e326ce064df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh" Oct 30 13:22:02.057828 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:02.057810 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgzcz\" (UniqueName: \"kubernetes.io/projected/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-kube-api-access-sgzcz\") pod \"maas-api-revoke-sub-d576e68b-5r8dh\" (UID: \"3cb49bf1-1cc0-4b74-b3cb-0e326ce064df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh" Oct 30 13:22:02.158986 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:02.158956 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-5r8dh\" (UID: \"3cb49bf1-1cc0-4b74-b3cb-0e326ce064df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh" Oct 30 13:22:02.159199 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:02.159007 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgzcz\" (UniqueName: \"kubernetes.io/projected/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-kube-api-access-sgzcz\") pod \"maas-api-revoke-sub-d576e68b-5r8dh\" (UID: \"3cb49bf1-1cc0-4b74-b3cb-0e326ce064df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh" Oct 30 13:22:02.159199 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:02.159035 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-5r8dh\" (UID: \"3cb49bf1-1cc0-4b74-b3cb-0e326ce064df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh" Oct 30 13:22:02.159654 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:02.159625 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-5r8dh\" (UID: \"3cb49bf1-1cc0-4b74-b3cb-0e326ce064df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh" Oct 30 13:22:02.161450 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:02.161428 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-5r8dh\" (UID: \"3cb49bf1-1cc0-4b74-b3cb-0e326ce064df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh" Oct 30 13:22:02.184733 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:02.184696 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgzcz\" (UniqueName: \"kubernetes.io/projected/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-kube-api-access-sgzcz\") pod \"maas-api-revoke-sub-d576e68b-5r8dh\" (UID: \"3cb49bf1-1cc0-4b74-b3cb-0e326ce064df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh" Oct 30 13:22:02.255092 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:02.254999 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh" Oct 30 13:22:02.363932 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:02.363893 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cc27019-ab9f-4475-bfc1-8d52bc737e62" path="/var/lib/kubelet/pods/2cc27019-ab9f-4475-bfc1-8d52bc737e62/volumes" Oct 30 13:22:02.562065 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:02.562014 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh"] Oct 30 13:22:02.564276 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:22:02.564240 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb49bf1_1cc0_4b74_b3cb_0e326ce064df.slice/crio-ba58b8eb549e2a39e67915d1a55033919c3e1b8aca9d87cfdbebc099aaa3c524 WatchSource:0}: Error finding container ba58b8eb549e2a39e67915d1a55033919c3e1b8aca9d87cfdbebc099aaa3c524: Status 404 returned error can't find the container with id ba58b8eb549e2a39e67915d1a55033919c3e1b8aca9d87cfdbebc099aaa3c524 Oct 30 13:22:02.613760 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:02.613722 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh" event={"ID":"3cb49bf1-1cc0-4b74-b3cb-0e326ce064df","Type":"ContainerStarted","Data":"ba58b8eb549e2a39e67915d1a55033919c3e1b8aca9d87cfdbebc099aaa3c524"} Oct 30 13:22:03.620572 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:03.620483 4503 generic.go:356] "Generic (PLEG): container finished" podID="3cb49bf1-1cc0-4b74-b3cb-0e326ce064df" containerID="da86d0df9963ddb545781eda70474a1309094ede557b98ab0f45ad57eff0b290" exitCode=0 Oct 30 13:22:03.620572 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:03.620534 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh" event={"ID":"3cb49bf1-1cc0-4b74-b3cb-0e326ce064df","Type":"ContainerDied","Data":"da86d0df9963ddb545781eda70474a1309094ede557b98ab0f45ad57eff0b290"} Oct 30 13:22:04.762272 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:04.762248 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh" Oct 30 13:22:04.886054 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:04.885913 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-maas-api-xks-ca\") pod \"3cb49bf1-1cc0-4b74-b3cb-0e326ce064df\" (UID: \"3cb49bf1-1cc0-4b74-b3cb-0e326ce064df\") " Oct 30 13:22:04.886054 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:04.886027 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-kube-api-access-sgzcz\" (UniqueName: \"kubernetes.io/projected/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-kube-api-access-sgzcz\") pod \"3cb49bf1-1cc0-4b74-b3cb-0e326ce064df\" (UID: \"3cb49bf1-1cc0-4b74-b3cb-0e326ce064df\") " Oct 30 13:22:04.886297 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:04.886101 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-maas-api-openshift-ca\") pod \"3cb49bf1-1cc0-4b74-b3cb-0e326ce064df\" (UID: \"3cb49bf1-1cc0-4b74-b3cb-0e326ce064df\") " Oct 30 13:22:04.886414 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:04.886389 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-maas-api-openshift-ca" pod "3cb49bf1-1cc0-4b74-b3cb-0e326ce064df" (UID: "3cb49bf1-1cc0-4b74-b3cb-0e326ce064df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:22:04.888215 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:04.888183 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-maas-api-xks-ca" pod "3cb49bf1-1cc0-4b74-b3cb-0e326ce064df" (UID: "3cb49bf1-1cc0-4b74-b3cb-0e326ce064df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:22:04.888332 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:04.888241 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-kube-api-access-sgzcz" pod "3cb49bf1-1cc0-4b74-b3cb-0e326ce064df" (UID: "3cb49bf1-1cc0-4b74-b3cb-0e326ce064df"). InnerVolumeSpecName "kube-api-access-sgzcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:22:04.987609 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:04.987569 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-maas-api-xks-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:22:04.987609 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:04.987603 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgzcz\" (UniqueName: \"kubernetes.io/projected/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-kube-api-access-sgzcz\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:22:04.987609 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:04.987614 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df-maas-api-openshift-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:22:05.633638 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:05.633605 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh" Oct 30 13:22:05.633638 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:05.633628 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh" event={"ID":"3cb49bf1-1cc0-4b74-b3cb-0e326ce064df","Type":"ContainerDied","Data":"ba58b8eb549e2a39e67915d1a55033919c3e1b8aca9d87cfdbebc099aaa3c524"} Oct 30 13:22:05.633856 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:05.633670 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba58b8eb549e2a39e67915d1a55033919c3e1b8aca9d87cfdbebc099aaa3c524" Oct 30 13:22:22.036733 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:22.036690 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd"] Oct 30 13:22:22.048659 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:22.048626 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j4hcd"] Oct 30 13:22:22.361141 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:22.361065 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="035c0081-10fb-487e-8881-dbd7f7a6aa33" path="/var/lib/kubelet/pods/035c0081-10fb-487e-8881-dbd7f7a6aa33/volumes" Oct 30 13:22:29.701063 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:22:29.700969 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:22:34.664044 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.664007 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb"] Oct 30 13:22:34.664481 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.664467 4503 cpu_manager.go:414] "containerMap: removing container" podUID="3cb49bf1-1cc0-4b74-b3cb-0e326ce064df" containerName="revoke-keys" Oct 30 13:22:34.664524 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.664482 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb49bf1-1cc0-4b74-b3cb-0e326ce064df" containerName="revoke-keys" Oct 30 13:22:34.664559 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.664544 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb49bf1-1cc0-4b74-b3cb-0e326ce064df" containerName="revoke-keys" Oct 30 13:22:34.668858 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.668837 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" Oct 30 13:22:34.691801 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.691764 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb"] Oct 30 13:22:34.699049 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.699021 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-2cc21d-serving-cert\"" Oct 30 13:22:34.699230 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.699070 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 13:22:34.699230 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.699176 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d5n24\"" Oct 30 13:22:34.747367 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.747322 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb\" (UID: \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" Oct 30 13:22:34.747583 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.747371 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnp8f\" (UniqueName: \"kubernetes.io/projected/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-kube-api-access-wnp8f\") pod \"maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb\" (UID: \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" Oct 30 13:22:34.747909 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.747579 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb\" (UID: \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" Oct 30 13:22:34.747909 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.747737 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb\" (UID: \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" Oct 30 13:22:34.848991 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.848950 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb\" (UID: \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" Oct 30 13:22:34.849189 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.849061 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb\" (UID: \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" Oct 30 13:22:34.849189 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.849117 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb\" (UID: \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" Oct 30 13:22:34.849189 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.849143 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnp8f\" (UniqueName: \"kubernetes.io/projected/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-kube-api-access-wnp8f\") pod \"maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb\" (UID: \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" Oct 30 13:22:34.849737 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.849709 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb\" (UID: \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" Oct 30 13:22:34.851459 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.851435 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb\" (UID: \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" Oct 30 13:22:34.851566 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.851537 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb\" (UID: \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" Oct 30 13:22:34.876762 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.876733 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnp8f\" (UniqueName: \"kubernetes.io/projected/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-kube-api-access-wnp8f\") pod \"maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb\" (UID: \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" Oct 30 13:22:34.983013 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:34.982917 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" Oct 30 13:22:35.331634 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:35.331605 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb"] Oct 30 13:22:35.331634 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:22:35.331616 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27bb7c31_f4d9_4d23_ac80_f5ffa8fa6bb9.slice/crio-4ef4b6f350f127bf3802aaa7e50eacfdc52d7d1e073a9464b2252e45065b5675 WatchSource:0}: Error finding container 4ef4b6f350f127bf3802aaa7e50eacfdc52d7d1e073a9464b2252e45065b5675: Status 404 returned error can't find the container with id 4ef4b6f350f127bf3802aaa7e50eacfdc52d7d1e073a9464b2252e45065b5675 Oct 30 13:22:35.332993 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:35.332973 4503 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:22:35.832981 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:35.832889 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" event={"ID":"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9","Type":"ContainerStarted","Data":"c1b565649ce64e11441ea2dc52283cdd7a168a7f36a3ea05d8c2989d38bfe596"} Oct 30 13:22:35.833456 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:35.833004 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" event={"ID":"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9","Type":"ContainerStarted","Data":"4ef4b6f350f127bf3802aaa7e50eacfdc52d7d1e073a9464b2252e45065b5675"} Oct 30 13:22:35.833456 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:35.833127 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" Oct 30 13:22:35.878246 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:35.878191 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" podStartSLOduration=1.496677181 podStartE2EDuration="1.878178103s" totalImagesPullingTime="381.500922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:35.875651464 +0000 UTC m=+1849.698879741" watchObservedRunningTime="2026-10-30 13:22:35.878178103 +0000 UTC m=+1849.701406381" Oct 30 13:22:38.034658 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:38.034619 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs"] Oct 30 13:22:38.046761 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:38.046731 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-v5mgs"] Oct 30 13:22:38.362833 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:38.362747 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a510b36a-5ce1-4047-8fcb-39d341e880c8" path="/var/lib/kubelet/pods/a510b36a-5ce1-4047-8fcb-39d341e880c8/volumes" Oct 30 13:22:41.844043 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:41.844009 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" Oct 30 13:22:47.909006 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:47.908969 4503 scope.go:122] "RemoveContainer" containerID="51e7647980b7c15d391168d68d827e9d3999674f91f907936bc6cdb494f1cee7" Oct 30 13:22:47.918050 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:47.918028 4503 scope.go:122] "RemoveContainer" containerID="bdb869baa937b324e11e3793e3bac6450d2681fa5e8794a08f169a571df42f00" Oct 30 13:22:47.926666 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:22:47.926646 4503 scope.go:122] "RemoveContainer" containerID="488838be2bf27fe0a391cda8669c0260b4e796d563c23e94365667b74d7b0e9e" Oct 30 13:22:59.701118 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:22:59.701077 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:23:29.700509 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:23:29.700470 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:23:33.797864 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:33.797831 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb"] Oct 30 13:23:33.798564 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:33.798071 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" podUID="27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9" containerName="maas-api" containerID="cri-o://c1b565649ce64e11441ea2dc52283cdd7a168a7f36a3ea05d8c2989d38bfe596" gracePeriod=30 Oct 30 13:23:35.057475 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.057447 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" Oct 30 13:23:35.122367 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.122336 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-metrics-certs\") pod \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\" (UID: \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\") " Oct 30 13:23:35.122551 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.122395 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-openshift-service-ca\") pod \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\" (UID: \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\") " Oct 30 13:23:35.122551 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.122447 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-maas-api-tls\") pod \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\" (UID: \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\") " Oct 30 13:23:35.122551 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.122491 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-kube-api-access-wnp8f\" (UniqueName: \"kubernetes.io/projected/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-kube-api-access-wnp8f\") pod \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\" (UID: \"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9\") " Oct 30 13:23:35.122827 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.122800 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-openshift-service-ca" pod "27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9" (UID: "27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:23:35.122917 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.122889 4503 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-openshift-service-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:23:35.124748 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.124722 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-maas-api-tls" pod "27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9" (UID: "27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:23:35.124966 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.124909 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-metrics-certs" pod "27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9" (UID: "27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:23:35.125101 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.125082 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-kube-api-access-wnp8f" pod "27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9" (UID: "27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9"). InnerVolumeSpecName "kube-api-access-wnp8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:35.205896 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.205791 4503 generic.go:356] "Generic (PLEG): container finished" podID="27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9" containerID="c1b565649ce64e11441ea2dc52283cdd7a168a7f36a3ea05d8c2989d38bfe596" exitCode=0 Oct 30 13:23:35.205896 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.205847 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" Oct 30 13:23:35.206127 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.205889 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" event={"ID":"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9","Type":"ContainerDied","Data":"c1b565649ce64e11441ea2dc52283cdd7a168a7f36a3ea05d8c2989d38bfe596"} Oct 30 13:23:35.206127 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.205993 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb" event={"ID":"27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9","Type":"ContainerDied","Data":"4ef4b6f350f127bf3802aaa7e50eacfdc52d7d1e073a9464b2252e45065b5675"} Oct 30 13:23:35.206127 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.206034 4503 scope.go:122] "RemoveContainer" containerID="c1b565649ce64e11441ea2dc52283cdd7a168a7f36a3ea05d8c2989d38bfe596" Oct 30 13:23:35.217188 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.217167 4503 scope.go:122] "RemoveContainer" containerID="c1b565649ce64e11441ea2dc52283cdd7a168a7f36a3ea05d8c2989d38bfe596" Oct 30 13:23:35.217459 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:23:35.217438 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b565649ce64e11441ea2dc52283cdd7a168a7f36a3ea05d8c2989d38bfe596\": container with ID starting with c1b565649ce64e11441ea2dc52283cdd7a168a7f36a3ea05d8c2989d38bfe596 not found: ID does not exist" containerID="c1b565649ce64e11441ea2dc52283cdd7a168a7f36a3ea05d8c2989d38bfe596" Oct 30 13:23:35.217522 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.217474 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b565649ce64e11441ea2dc52283cdd7a168a7f36a3ea05d8c2989d38bfe596"} err="failed to get container status \"c1b565649ce64e11441ea2dc52283cdd7a168a7f36a3ea05d8c2989d38bfe596\": rpc error: code = NotFound desc = could not find container \"c1b565649ce64e11441ea2dc52283cdd7a168a7f36a3ea05d8c2989d38bfe596\": container with ID starting with c1b565649ce64e11441ea2dc52283cdd7a168a7f36a3ea05d8c2989d38bfe596 not found: ID does not exist" Oct 30 13:23:35.224372 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.224344 4503 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-metrics-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:23:35.224372 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.224368 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-maas-api-tls\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:23:35.224372 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.224378 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnp8f\" (UniqueName: \"kubernetes.io/projected/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9-kube-api-access-wnp8f\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:23:35.261046 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.261004 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb"] Oct 30 13:23:35.266194 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:35.266162 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2cc21d-85cbf45b8b-flqpb"] Oct 30 13:23:36.393516 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:36.393478 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9" path="/var/lib/kubelet/pods/27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9/volumes" Oct 30 13:23:47.956566 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:47.956529 4503 scope.go:122] "RemoveContainer" containerID="bea88ae43fb4cb3e11a018d677ae4c9fe32b5631ade147bb73782549c13d0391" Oct 30 13:23:47.965479 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:47.965456 4503 scope.go:122] "RemoveContainer" containerID="8b6839f2114dacaa9faca439266d107977051e9b05681cadd5481d59f92206e2" Oct 30 13:23:47.976474 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:47.976447 4503 scope.go:122] "RemoveContainer" containerID="25b846dca2bf694160e50d5d67eac84fa0e3d2e9a4281cc5160d5f314cb1b7be" Oct 30 13:23:47.984361 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:47.984340 4503 scope.go:122] "RemoveContainer" containerID="5444117418b9225a470c9c3010cbb8df9d087ec062ed628ba56d6ab558cdc500" Oct 30 13:23:52.380963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:52.380116 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8ndvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:23:52.380963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:52.380200 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" podUID="d4d5e227-0a76-4cb2-b460-8409c61a9f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:23:52.443594 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:52.443144 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gdg2m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:23:52.443594 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:23:52.443238 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" podUID="f6fc3d60-979c-4a16-9834-7c1083464b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:23:59.700846 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:23:59.700757 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:24:21.886483 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:21.886433 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz"] Oct 30 13:24:21.887230 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:21.887198 4503 cpu_manager.go:414] "containerMap: removing container" podUID="27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9" containerName="maas-api" Oct 30 13:24:21.887230 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:21.887226 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9" containerName="maas-api" Oct 30 13:24:21.887431 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:21.887355 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="27bb7c31-f4d9-4d23-ac80-f5ffa8fa6bb9" containerName="maas-api" Oct 30 13:24:21.891275 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:21.891253 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz" Oct 30 13:24:21.916080 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:21.916048 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz"] Oct 30 13:24:21.923499 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:21.923475 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:24:21.923860 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:21.923840 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:24:21.998784 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:21.998751 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1267dbda-9b9c-4e67-842e-6b5a03c32650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-d64vz\" (UID: \"1267dbda-9b9c-4e67-842e-6b5a03c32650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz" Oct 30 13:24:21.999276 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:21.999233 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1267dbda-9b9c-4e67-842e-6b5a03c32650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-d64vz\" (UID: \"1267dbda-9b9c-4e67-842e-6b5a03c32650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz" Oct 30 13:24:21.999498 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:21.999478 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bs5r\" (UniqueName: \"kubernetes.io/projected/1267dbda-9b9c-4e67-842e-6b5a03c32650-kube-api-access-9bs5r\") pod \"maas-api-revoke-sub-452b6d7b-d64vz\" (UID: \"1267dbda-9b9c-4e67-842e-6b5a03c32650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz" Oct 30 13:24:22.101328 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:22.101270 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1267dbda-9b9c-4e67-842e-6b5a03c32650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-d64vz\" (UID: \"1267dbda-9b9c-4e67-842e-6b5a03c32650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz" Oct 30 13:24:22.101540 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:22.101473 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1267dbda-9b9c-4e67-842e-6b5a03c32650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-d64vz\" (UID: \"1267dbda-9b9c-4e67-842e-6b5a03c32650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz" Oct 30 13:24:22.101540 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:22.101520 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bs5r\" (UniqueName: \"kubernetes.io/projected/1267dbda-9b9c-4e67-842e-6b5a03c32650-kube-api-access-9bs5r\") pod \"maas-api-revoke-sub-452b6d7b-d64vz\" (UID: \"1267dbda-9b9c-4e67-842e-6b5a03c32650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz" Oct 30 13:24:22.102364 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:22.102337 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1267dbda-9b9c-4e67-842e-6b5a03c32650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-d64vz\" (UID: \"1267dbda-9b9c-4e67-842e-6b5a03c32650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz" Oct 30 13:24:22.104503 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:22.104476 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1267dbda-9b9c-4e67-842e-6b5a03c32650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-d64vz\" (UID: \"1267dbda-9b9c-4e67-842e-6b5a03c32650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz" Oct 30 13:24:22.127829 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:22.127792 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bs5r\" (UniqueName: \"kubernetes.io/projected/1267dbda-9b9c-4e67-842e-6b5a03c32650-kube-api-access-9bs5r\") pod \"maas-api-revoke-sub-452b6d7b-d64vz\" (UID: \"1267dbda-9b9c-4e67-842e-6b5a03c32650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz" Oct 30 13:24:22.228438 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:22.228396 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz" Oct 30 13:24:22.637529 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:22.637487 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz"] Oct 30 13:24:23.605963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:23.591111 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz" event={"ID":"1267dbda-9b9c-4e67-842e-6b5a03c32650","Type":"ContainerStarted","Data":"f5069a5ff7ca5fdb87eb2b468acd79a03e34fb87e96e30caed2c423c2b812204"} Oct 30 13:24:23.605963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:23.591183 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz" event={"ID":"1267dbda-9b9c-4e67-842e-6b5a03c32650","Type":"ContainerStarted","Data":"5c0557518b0f353367670e2fb9a0c8f0a507b85bbfda7508aadc333c1aa88a89"} Oct 30 13:24:24.615963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:24.615363 4503 generic.go:356] "Generic (PLEG): container finished" podID="1267dbda-9b9c-4e67-842e-6b5a03c32650" containerID="f5069a5ff7ca5fdb87eb2b468acd79a03e34fb87e96e30caed2c423c2b812204" exitCode=0 Oct 30 13:24:24.615963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:24.615456 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz" event={"ID":"1267dbda-9b9c-4e67-842e-6b5a03c32650","Type":"ContainerDied","Data":"f5069a5ff7ca5fdb87eb2b468acd79a03e34fb87e96e30caed2c423c2b812204"} Oct 30 13:24:25.758373 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:25.758344 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz" Oct 30 13:24:25.849066 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:25.849025 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1267dbda-9b9c-4e67-842e-6b5a03c32650-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1267dbda-9b9c-4e67-842e-6b5a03c32650-maas-api-openshift-ca\") pod \"1267dbda-9b9c-4e67-842e-6b5a03c32650\" (UID: \"1267dbda-9b9c-4e67-842e-6b5a03c32650\") " Oct 30 13:24:25.849259 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:25.849117 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1267dbda-9b9c-4e67-842e-6b5a03c32650-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1267dbda-9b9c-4e67-842e-6b5a03c32650-maas-api-xks-ca\") pod \"1267dbda-9b9c-4e67-842e-6b5a03c32650\" (UID: \"1267dbda-9b9c-4e67-842e-6b5a03c32650\") " Oct 30 13:24:25.849259 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:25.849145 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1267dbda-9b9c-4e67-842e-6b5a03c32650-kube-api-access-9bs5r\" (UniqueName: \"kubernetes.io/projected/1267dbda-9b9c-4e67-842e-6b5a03c32650-kube-api-access-9bs5r\") pod \"1267dbda-9b9c-4e67-842e-6b5a03c32650\" (UID: \"1267dbda-9b9c-4e67-842e-6b5a03c32650\") " Oct 30 13:24:25.849455 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:25.849423 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1267dbda-9b9c-4e67-842e-6b5a03c32650-maas-api-openshift-ca" pod "1267dbda-9b9c-4e67-842e-6b5a03c32650" (UID: "1267dbda-9b9c-4e67-842e-6b5a03c32650"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:25.851246 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:25.851222 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1267dbda-9b9c-4e67-842e-6b5a03c32650-maas-api-xks-ca" pod "1267dbda-9b9c-4e67-842e-6b5a03c32650" (UID: "1267dbda-9b9c-4e67-842e-6b5a03c32650"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:24:25.851334 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:25.851264 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1267dbda-9b9c-4e67-842e-6b5a03c32650-kube-api-access-9bs5r" pod "1267dbda-9b9c-4e67-842e-6b5a03c32650" (UID: "1267dbda-9b9c-4e67-842e-6b5a03c32650"). InnerVolumeSpecName "kube-api-access-9bs5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:25.950459 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:25.950419 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1267dbda-9b9c-4e67-842e-6b5a03c32650-maas-api-openshift-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:24:25.950459 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:25.950455 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1267dbda-9b9c-4e67-842e-6b5a03c32650-maas-api-xks-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:24:25.950459 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:25.950464 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bs5r\" (UniqueName: \"kubernetes.io/projected/1267dbda-9b9c-4e67-842e-6b5a03c32650-kube-api-access-9bs5r\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:24:26.627452 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:26.627358 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz" event={"ID":"1267dbda-9b9c-4e67-842e-6b5a03c32650","Type":"ContainerDied","Data":"5c0557518b0f353367670e2fb9a0c8f0a507b85bbfda7508aadc333c1aa88a89"} Oct 30 13:24:26.627452 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:26.627393 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d64vz" Oct 30 13:24:26.627452 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:26.627414 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c0557518b0f353367670e2fb9a0c8f0a507b85bbfda7508aadc333c1aa88a89" Oct 30 13:24:29.702975 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:24:29.702909 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:24:42.377702 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:42.377659 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd"] Oct 30 13:24:42.378374 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:42.378350 4503 cpu_manager.go:414] "containerMap: removing container" podUID="1267dbda-9b9c-4e67-842e-6b5a03c32650" containerName="revoke-keys" Oct 30 13:24:42.378472 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:42.378377 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1267dbda-9b9c-4e67-842e-6b5a03c32650" containerName="revoke-keys" Oct 30 13:24:42.378540 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:42.378501 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="1267dbda-9b9c-4e67-842e-6b5a03c32650" containerName="revoke-keys" Oct 30 13:24:42.382160 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:42.382137 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd" Oct 30 13:24:42.411612 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:42.404480 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd"] Oct 30 13:24:42.411612 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:42.410376 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:24:42.411612 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:42.410506 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29wbb\"" Oct 30 13:24:42.486630 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:42.486573 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844ba902-8fbc-4051-9872-32ec7a2fc96d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-c4djd\" (UID: \"844ba902-8fbc-4051-9872-32ec7a2fc96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd" Oct 30 13:24:42.486831 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:42.486640 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844ba902-8fbc-4051-9872-32ec7a2fc96d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-c4djd\" (UID: \"844ba902-8fbc-4051-9872-32ec7a2fc96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd" Oct 30 13:24:42.486831 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:42.486669 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs9hn\" (UniqueName: \"kubernetes.io/projected/844ba902-8fbc-4051-9872-32ec7a2fc96d-kube-api-access-gs9hn\") pod \"maas-api-revoke-sub-3e31a506-c4djd\" (UID: \"844ba902-8fbc-4051-9872-32ec7a2fc96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd" Oct 30 13:24:42.587517 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:42.587466 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844ba902-8fbc-4051-9872-32ec7a2fc96d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-c4djd\" (UID: \"844ba902-8fbc-4051-9872-32ec7a2fc96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd" Oct 30 13:24:42.587731 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:42.587529 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844ba902-8fbc-4051-9872-32ec7a2fc96d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-c4djd\" (UID: \"844ba902-8fbc-4051-9872-32ec7a2fc96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd" Oct 30 13:24:42.587731 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:42.587561 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs9hn\" (UniqueName: \"kubernetes.io/projected/844ba902-8fbc-4051-9872-32ec7a2fc96d-kube-api-access-gs9hn\") pod \"maas-api-revoke-sub-3e31a506-c4djd\" (UID: \"844ba902-8fbc-4051-9872-32ec7a2fc96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd" Oct 30 13:24:42.589292 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:42.589266 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844ba902-8fbc-4051-9872-32ec7a2fc96d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-c4djd\" (UID: \"844ba902-8fbc-4051-9872-32ec7a2fc96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd" Oct 30 13:24:42.591077 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:42.591046 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844ba902-8fbc-4051-9872-32ec7a2fc96d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-c4djd\" (UID: \"844ba902-8fbc-4051-9872-32ec7a2fc96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd" Oct 30 13:24:42.614743 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:42.614707 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs9hn\" (UniqueName: \"kubernetes.io/projected/844ba902-8fbc-4051-9872-32ec7a2fc96d-kube-api-access-gs9hn\") pod \"maas-api-revoke-sub-3e31a506-c4djd\" (UID: \"844ba902-8fbc-4051-9872-32ec7a2fc96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd" Oct 30 13:24:42.703956 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:42.703897 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd" Oct 30 13:24:43.044102 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:43.044013 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd"] Oct 30 13:24:43.049194 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:24:43.049152 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844ba902_8fbc_4051_9872_32ec7a2fc96d.slice/crio-c02381183d2a6c4cb9825d1e54cd4db2da858e64642014d49ad626f3be2d2f7f WatchSource:0}: Error finding container c02381183d2a6c4cb9825d1e54cd4db2da858e64642014d49ad626f3be2d2f7f: Status 404 returned error can't find the container with id c02381183d2a6c4cb9825d1e54cd4db2da858e64642014d49ad626f3be2d2f7f Oct 30 13:24:43.896657 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:43.895679 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd" event={"ID":"844ba902-8fbc-4051-9872-32ec7a2fc96d","Type":"ContainerStarted","Data":"15b5ab57543652c35083aac5aa64950df512281400ac2ab1b37362d34f671f96"} Oct 30 13:24:43.896657 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:43.895751 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd" event={"ID":"844ba902-8fbc-4051-9872-32ec7a2fc96d","Type":"ContainerStarted","Data":"c02381183d2a6c4cb9825d1e54cd4db2da858e64642014d49ad626f3be2d2f7f"} Oct 30 13:24:44.932960 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:44.932568 4503 generic.go:356] "Generic (PLEG): container finished" podID="844ba902-8fbc-4051-9872-32ec7a2fc96d" containerID="15b5ab57543652c35083aac5aa64950df512281400ac2ab1b37362d34f671f96" exitCode=0 Oct 30 13:24:44.932960 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:44.932710 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd" event={"ID":"844ba902-8fbc-4051-9872-32ec7a2fc96d","Type":"ContainerDied","Data":"15b5ab57543652c35083aac5aa64950df512281400ac2ab1b37362d34f671f96"} Oct 30 13:24:46.079155 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:46.079126 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd" Oct 30 13:24:46.269169 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:46.269087 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/844ba902-8fbc-4051-9872-32ec7a2fc96d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844ba902-8fbc-4051-9872-32ec7a2fc96d-maas-api-xks-ca\") pod \"844ba902-8fbc-4051-9872-32ec7a2fc96d\" (UID: \"844ba902-8fbc-4051-9872-32ec7a2fc96d\") " Oct 30 13:24:46.269169 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:46.269140 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844ba902-8fbc-4051-9872-32ec7a2fc96d-kube-api-access-gs9hn\" (UniqueName: \"kubernetes.io/projected/844ba902-8fbc-4051-9872-32ec7a2fc96d-kube-api-access-gs9hn\") pod \"844ba902-8fbc-4051-9872-32ec7a2fc96d\" (UID: \"844ba902-8fbc-4051-9872-32ec7a2fc96d\") " Oct 30 13:24:46.269390 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:46.269173 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844ba902-8fbc-4051-9872-32ec7a2fc96d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844ba902-8fbc-4051-9872-32ec7a2fc96d-maas-api-openshift-ca\") pod \"844ba902-8fbc-4051-9872-32ec7a2fc96d\" (UID: \"844ba902-8fbc-4051-9872-32ec7a2fc96d\") " Oct 30 13:24:46.269611 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:46.269586 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844ba902-8fbc-4051-9872-32ec7a2fc96d-maas-api-openshift-ca" pod "844ba902-8fbc-4051-9872-32ec7a2fc96d" (UID: "844ba902-8fbc-4051-9872-32ec7a2fc96d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:46.271194 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:46.271166 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/844ba902-8fbc-4051-9872-32ec7a2fc96d-maas-api-xks-ca" pod "844ba902-8fbc-4051-9872-32ec7a2fc96d" (UID: "844ba902-8fbc-4051-9872-32ec7a2fc96d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:24:46.271305 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:46.271264 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844ba902-8fbc-4051-9872-32ec7a2fc96d-kube-api-access-gs9hn" pod "844ba902-8fbc-4051-9872-32ec7a2fc96d" (UID: "844ba902-8fbc-4051-9872-32ec7a2fc96d"). InnerVolumeSpecName "kube-api-access-gs9hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:46.370259 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:46.370111 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844ba902-8fbc-4051-9872-32ec7a2fc96d-maas-api-xks-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:24:46.370259 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:46.370138 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs9hn\" (UniqueName: \"kubernetes.io/projected/844ba902-8fbc-4051-9872-32ec7a2fc96d-kube-api-access-gs9hn\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:24:46.370259 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:46.370151 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844ba902-8fbc-4051-9872-32ec7a2fc96d-maas-api-openshift-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:24:46.946011 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:46.945972 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd" event={"ID":"844ba902-8fbc-4051-9872-32ec7a2fc96d","Type":"ContainerDied","Data":"c02381183d2a6c4cb9825d1e54cd4db2da858e64642014d49ad626f3be2d2f7f"} Oct 30 13:24:46.946225 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:46.946034 4503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c02381183d2a6c4cb9825d1e54cd4db2da858e64642014d49ad626f3be2d2f7f" Oct 30 13:24:46.946225 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:46.945980 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-c4djd" Oct 30 13:24:48.019827 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:24:48.019792 4503 scope.go:122] "RemoveContainer" containerID="83539a94cc4887314505dbf06d178fd7ee090e7d5153ba7ec0f1166fc3752040" Oct 30 13:24:59.819437 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:24:59.819375 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:25:16.047155 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:16.047113 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q"] Oct 30 13:25:16.059680 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:16.059642 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x858q"] Oct 30 13:25:16.362022 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:16.361905 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b846c45-29d4-4995-b2a5-98fdc1f20954" path="/var/lib/kubelet/pods/8b846c45-29d4-4995-b2a5-98fdc1f20954/volumes" Oct 30 13:25:17.476967 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.476903 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-96ll8"] Oct 30 13:25:17.477416 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.477402 4503 cpu_manager.go:414] "containerMap: removing container" podUID="844ba902-8fbc-4051-9872-32ec7a2fc96d" containerName="revoke-keys" Oct 30 13:25:17.477466 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.477417 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844ba902-8fbc-4051-9872-32ec7a2fc96d" containerName="revoke-keys" Oct 30 13:25:17.477519 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.477503 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="844ba902-8fbc-4051-9872-32ec7a2fc96d" containerName="revoke-keys" Oct 30 13:25:17.480805 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.480782 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" Oct 30 13:25:17.499247 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.499215 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-9g547"] Oct 30 13:25:17.502654 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.502630 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-9g547" Oct 30 13:25:17.509030 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.509005 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 13:25:17.509280 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.509069 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 13:25:17.509423 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.509069 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d5n24\"" Oct 30 13:25:17.513219 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.513194 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-96ll8"] Oct 30 13:25:17.523646 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.523606 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6526146-a37b-4fae-94ab-53e7c749c66d-maas-api-tls\") pod \"maas-api-5c48877c75-96ll8\" (UID: \"a6526146-a37b-4fae-94ab-53e7c749c66d\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" Oct 30 13:25:17.523918 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.523890 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6526146-a37b-4fae-94ab-53e7c749c66d-openshift-service-ca\") pod \"maas-api-5c48877c75-96ll8\" (UID: \"a6526146-a37b-4fae-94ab-53e7c749c66d\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" Oct 30 13:25:17.524087 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.523986 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6526146-a37b-4fae-94ab-53e7c749c66d-metrics-certs\") pod \"maas-api-5c48877c75-96ll8\" (UID: \"a6526146-a37b-4fae-94ab-53e7c749c66d\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" Oct 30 13:25:17.524087 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.524068 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjmcj\" (UniqueName: \"kubernetes.io/projected/a6526146-a37b-4fae-94ab-53e7c749c66d-kube-api-access-fjmcj\") pod \"maas-api-5c48877c75-96ll8\" (UID: \"a6526146-a37b-4fae-94ab-53e7c749c66d\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" Oct 30 13:25:17.526359 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.526333 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-9g547"] Oct 30 13:25:17.624969 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.624907 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6526146-a37b-4fae-94ab-53e7c749c66d-maas-api-tls\") pod \"maas-api-5c48877c75-96ll8\" (UID: \"a6526146-a37b-4fae-94ab-53e7c749c66d\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" Oct 30 13:25:17.625164 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.624984 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6526146-a37b-4fae-94ab-53e7c749c66d-openshift-service-ca\") pod \"maas-api-5c48877c75-96ll8\" (UID: \"a6526146-a37b-4fae-94ab-53e7c749c66d\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" Oct 30 13:25:17.625164 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.625018 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff77317-aae4-46c5-8e69-b5ede8f43cdf-metrics-certs\") pod \"maas-api-595cc648c-9g547\" (UID: \"6ff77317-aae4-46c5-8e69-b5ede8f43cdf\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9g547" Oct 30 13:25:17.625164 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.625065 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6526146-a37b-4fae-94ab-53e7c749c66d-metrics-certs\") pod \"maas-api-5c48877c75-96ll8\" (UID: \"a6526146-a37b-4fae-94ab-53e7c749c66d\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" Oct 30 13:25:17.625164 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.625142 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ff77317-aae4-46c5-8e69-b5ede8f43cdf-maas-api-tls\") pod \"maas-api-595cc648c-9g547\" (UID: \"6ff77317-aae4-46c5-8e69-b5ede8f43cdf\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9g547" Oct 30 13:25:17.625395 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.625217 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff77317-aae4-46c5-8e69-b5ede8f43cdf-openshift-service-ca\") pod \"maas-api-595cc648c-9g547\" (UID: \"6ff77317-aae4-46c5-8e69-b5ede8f43cdf\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9g547" Oct 30 13:25:17.625395 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.625284 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjmcj\" (UniqueName: \"kubernetes.io/projected/a6526146-a37b-4fae-94ab-53e7c749c66d-kube-api-access-fjmcj\") pod \"maas-api-5c48877c75-96ll8\" (UID: \"a6526146-a37b-4fae-94ab-53e7c749c66d\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" Oct 30 13:25:17.625395 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.625311 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htwsl\" (UniqueName: \"kubernetes.io/projected/6ff77317-aae4-46c5-8e69-b5ede8f43cdf-kube-api-access-htwsl\") pod \"maas-api-595cc648c-9g547\" (UID: \"6ff77317-aae4-46c5-8e69-b5ede8f43cdf\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9g547" Oct 30 13:25:17.625900 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.625877 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6526146-a37b-4fae-94ab-53e7c749c66d-openshift-service-ca\") pod \"maas-api-5c48877c75-96ll8\" (UID: \"a6526146-a37b-4fae-94ab-53e7c749c66d\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" Oct 30 13:25:17.627476 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.627445 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6526146-a37b-4fae-94ab-53e7c749c66d-metrics-certs\") pod \"maas-api-5c48877c75-96ll8\" (UID: \"a6526146-a37b-4fae-94ab-53e7c749c66d\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" Oct 30 13:25:17.627567 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.627521 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6526146-a37b-4fae-94ab-53e7c749c66d-maas-api-tls\") pod \"maas-api-5c48877c75-96ll8\" (UID: \"a6526146-a37b-4fae-94ab-53e7c749c66d\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" Oct 30 13:25:17.651319 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.651286 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjmcj\" (UniqueName: \"kubernetes.io/projected/a6526146-a37b-4fae-94ab-53e7c749c66d-kube-api-access-fjmcj\") pod \"maas-api-5c48877c75-96ll8\" (UID: \"a6526146-a37b-4fae-94ab-53e7c749c66d\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" Oct 30 13:25:17.726339 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.726302 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff77317-aae4-46c5-8e69-b5ede8f43cdf-metrics-certs\") pod \"maas-api-595cc648c-9g547\" (UID: \"6ff77317-aae4-46c5-8e69-b5ede8f43cdf\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9g547" Oct 30 13:25:17.726532 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.726356 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ff77317-aae4-46c5-8e69-b5ede8f43cdf-maas-api-tls\") pod \"maas-api-595cc648c-9g547\" (UID: \"6ff77317-aae4-46c5-8e69-b5ede8f43cdf\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9g547" Oct 30 13:25:17.726598 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.726530 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff77317-aae4-46c5-8e69-b5ede8f43cdf-openshift-service-ca\") pod \"maas-api-595cc648c-9g547\" (UID: \"6ff77317-aae4-46c5-8e69-b5ede8f43cdf\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9g547" Oct 30 13:25:17.726656 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.726596 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htwsl\" (UniqueName: \"kubernetes.io/projected/6ff77317-aae4-46c5-8e69-b5ede8f43cdf-kube-api-access-htwsl\") pod \"maas-api-595cc648c-9g547\" (UID: \"6ff77317-aae4-46c5-8e69-b5ede8f43cdf\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9g547" Oct 30 13:25:17.727330 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.727269 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff77317-aae4-46c5-8e69-b5ede8f43cdf-openshift-service-ca\") pod \"maas-api-595cc648c-9g547\" (UID: \"6ff77317-aae4-46c5-8e69-b5ede8f43cdf\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9g547" Oct 30 13:25:17.728910 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.728888 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff77317-aae4-46c5-8e69-b5ede8f43cdf-metrics-certs\") pod \"maas-api-595cc648c-9g547\" (UID: \"6ff77317-aae4-46c5-8e69-b5ede8f43cdf\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9g547" Oct 30 13:25:17.729194 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.729171 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ff77317-aae4-46c5-8e69-b5ede8f43cdf-maas-api-tls\") pod \"maas-api-595cc648c-9g547\" (UID: \"6ff77317-aae4-46c5-8e69-b5ede8f43cdf\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9g547" Oct 30 13:25:17.754106 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.754073 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htwsl\" (UniqueName: \"kubernetes.io/projected/6ff77317-aae4-46c5-8e69-b5ede8f43cdf-kube-api-access-htwsl\") pod \"maas-api-595cc648c-9g547\" (UID: \"6ff77317-aae4-46c5-8e69-b5ede8f43cdf\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9g547" Oct 30 13:25:17.795126 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.795094 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" Oct 30 13:25:17.820857 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:17.820826 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-9g547" Oct 30 13:25:18.101055 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:18.100974 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-96ll8"] Oct 30 13:25:18.144896 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:18.144864 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-9g547"] Oct 30 13:25:18.145482 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:25:18.145453 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff77317_aae4_46c5_8e69_b5ede8f43cdf.slice/crio-895148c43f2e36661484516c7521997fdc15caa605e670077509fb932683ecba WatchSource:0}: Error finding container 895148c43f2e36661484516c7521997fdc15caa605e670077509fb932683ecba: Status 404 returned error can't find the container with id 895148c43f2e36661484516c7521997fdc15caa605e670077509fb932683ecba Oct 30 13:25:18.292459 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:18.292424 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" event={"ID":"a6526146-a37b-4fae-94ab-53e7c749c66d","Type":"ContainerStarted","Data":"1d166fde16dfd732f274df56446b1d410df10321e7b1fe4da5109951aad84d3a"} Oct 30 13:25:18.293591 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:18.293564 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-9g547" event={"ID":"6ff77317-aae4-46c5-8e69-b5ede8f43cdf","Type":"ContainerStarted","Data":"895148c43f2e36661484516c7521997fdc15caa605e670077509fb932683ecba"} Oct 30 13:25:19.300864 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:19.300818 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" event={"ID":"a6526146-a37b-4fae-94ab-53e7c749c66d","Type":"ContainerStarted","Data":"459986edc4f0c3d124d33162a93bfdf63e9d4ba0457cec13b5809649e5e25ced"} Oct 30 13:25:19.301374 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:19.300889 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" Oct 30 13:25:19.302597 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:19.302565 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-9g547" event={"ID":"6ff77317-aae4-46c5-8e69-b5ede8f43cdf","Type":"ContainerStarted","Data":"d932fc0ffdccc374a82323324d701d05596fa63e8d9f809cb12a0b42436f36b3"} Oct 30 13:25:19.302725 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:19.302633 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-9g547" Oct 30 13:25:19.344692 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:19.344610 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" podStartSLOduration=1.944201025 podStartE2EDuration="2.344592045s" totalImagesPullingTime="400.39102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:19.34316611 +0000 UTC m=+2013.166394384" watchObservedRunningTime="2026-10-30 13:25:19.344592045 +0000 UTC m=+2013.167820323" Oct 30 13:25:19.388012 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:19.387952 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-9g547" podStartSLOduration=2.07315234 podStartE2EDuration="2.38792383s" totalImagesPullingTime="314.77149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:19.386101757 +0000 UTC m=+2013.209330034" watchObservedRunningTime="2026-10-30 13:25:19.38792383 +0000 UTC m=+2013.211152107" Oct 30 13:25:25.313738 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:25.313703 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" Oct 30 13:25:25.314208 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:25.314105 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-9g547" Oct 30 13:25:25.434708 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:25.434679 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-96ll8"] Oct 30 13:25:25.434875 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:25.434852 4503 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" podUID="a6526146-a37b-4fae-94ab-53e7c749c66d" containerName="maas-api" containerID="cri-o://459986edc4f0c3d124d33162a93bfdf63e9d4ba0457cec13b5809649e5e25ced" gracePeriod=30 Oct 30 13:25:25.688677 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:25.688649 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" Oct 30 13:25:25.808726 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:25.808691 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6526146-a37b-4fae-94ab-53e7c749c66d-kube-api-access-fjmcj\" (UniqueName: \"kubernetes.io/projected/a6526146-a37b-4fae-94ab-53e7c749c66d-kube-api-access-fjmcj\") pod \"a6526146-a37b-4fae-94ab-53e7c749c66d\" (UID: \"a6526146-a37b-4fae-94ab-53e7c749c66d\") " Oct 30 13:25:25.808928 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:25.808772 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6526146-a37b-4fae-94ab-53e7c749c66d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6526146-a37b-4fae-94ab-53e7c749c66d-maas-api-tls\") pod \"a6526146-a37b-4fae-94ab-53e7c749c66d\" (UID: \"a6526146-a37b-4fae-94ab-53e7c749c66d\") " Oct 30 13:25:25.808928 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:25.808794 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6526146-a37b-4fae-94ab-53e7c749c66d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6526146-a37b-4fae-94ab-53e7c749c66d-openshift-service-ca\") pod \"a6526146-a37b-4fae-94ab-53e7c749c66d\" (UID: \"a6526146-a37b-4fae-94ab-53e7c749c66d\") " Oct 30 13:25:25.808928 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:25.808876 4503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6526146-a37b-4fae-94ab-53e7c749c66d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6526146-a37b-4fae-94ab-53e7c749c66d-metrics-certs\") pod \"a6526146-a37b-4fae-94ab-53e7c749c66d\" (UID: \"a6526146-a37b-4fae-94ab-53e7c749c66d\") " Oct 30 13:25:25.809341 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:25.809306 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6526146-a37b-4fae-94ab-53e7c749c66d-openshift-service-ca" pod "a6526146-a37b-4fae-94ab-53e7c749c66d" (UID: "a6526146-a37b-4fae-94ab-53e7c749c66d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:25.810977 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:25.810931 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6526146-a37b-4fae-94ab-53e7c749c66d-maas-api-tls" pod "a6526146-a37b-4fae-94ab-53e7c749c66d" (UID: "a6526146-a37b-4fae-94ab-53e7c749c66d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:25.811087 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:25.810992 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6526146-a37b-4fae-94ab-53e7c749c66d-metrics-certs" pod "a6526146-a37b-4fae-94ab-53e7c749c66d" (UID: "a6526146-a37b-4fae-94ab-53e7c749c66d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:25.811087 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:25.811069 4503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6526146-a37b-4fae-94ab-53e7c749c66d-kube-api-access-fjmcj" pod "a6526146-a37b-4fae-94ab-53e7c749c66d" (UID: "a6526146-a37b-4fae-94ab-53e7c749c66d"). InnerVolumeSpecName "kube-api-access-fjmcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:25.910850 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:25.910816 4503 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6526146-a37b-4fae-94ab-53e7c749c66d-metrics-certs\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:25:25.910850 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:25.910849 4503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjmcj\" (UniqueName: \"kubernetes.io/projected/a6526146-a37b-4fae-94ab-53e7c749c66d-kube-api-access-fjmcj\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:25:25.911089 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:25.910860 4503 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6526146-a37b-4fae-94ab-53e7c749c66d-maas-api-tls\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:25:25.911089 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:25.910869 4503 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6526146-a37b-4fae-94ab-53e7c749c66d-openshift-service-ca\") on node \"ip-10-0-130-200.ec2.internal\" DevicePath \"\"" Oct 30 13:25:26.345192 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:26.345091 4503 generic.go:356] "Generic (PLEG): container finished" podID="a6526146-a37b-4fae-94ab-53e7c749c66d" containerID="459986edc4f0c3d124d33162a93bfdf63e9d4ba0457cec13b5809649e5e25ced" exitCode=0 Oct 30 13:25:26.345192 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:26.345153 4503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" Oct 30 13:25:26.345745 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:26.345191 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" event={"ID":"a6526146-a37b-4fae-94ab-53e7c749c66d","Type":"ContainerDied","Data":"459986edc4f0c3d124d33162a93bfdf63e9d4ba0457cec13b5809649e5e25ced"} Oct 30 13:25:26.345745 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:26.345258 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-96ll8" event={"ID":"a6526146-a37b-4fae-94ab-53e7c749c66d","Type":"ContainerDied","Data":"1d166fde16dfd732f274df56446b1d410df10321e7b1fe4da5109951aad84d3a"} Oct 30 13:25:26.345745 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:26.345290 4503 scope.go:122] "RemoveContainer" containerID="459986edc4f0c3d124d33162a93bfdf63e9d4ba0457cec13b5809649e5e25ced" Oct 30 13:25:26.362032 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:26.361914 4503 scope.go:122] "RemoveContainer" containerID="459986edc4f0c3d124d33162a93bfdf63e9d4ba0457cec13b5809649e5e25ced" Oct 30 13:25:26.362377 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:25:26.362346 4503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"459986edc4f0c3d124d33162a93bfdf63e9d4ba0457cec13b5809649e5e25ced\": container with ID starting with 459986edc4f0c3d124d33162a93bfdf63e9d4ba0457cec13b5809649e5e25ced not found: ID does not exist" containerID="459986edc4f0c3d124d33162a93bfdf63e9d4ba0457cec13b5809649e5e25ced" Oct 30 13:25:26.362500 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:26.362390 4503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"459986edc4f0c3d124d33162a93bfdf63e9d4ba0457cec13b5809649e5e25ced"} err="failed to get container status \"459986edc4f0c3d124d33162a93bfdf63e9d4ba0457cec13b5809649e5e25ced\": rpc error: code = NotFound desc = could not find container \"459986edc4f0c3d124d33162a93bfdf63e9d4ba0457cec13b5809649e5e25ced\": container with ID starting with 459986edc4f0c3d124d33162a93bfdf63e9d4ba0457cec13b5809649e5e25ced not found: ID does not exist" Oct 30 13:25:26.410158 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:26.410112 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-96ll8"] Oct 30 13:25:26.421284 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:26.420784 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-96ll8"] Oct 30 13:25:28.401962 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:28.401518 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6526146-a37b-4fae-94ab-53e7c749c66d" path="/var/lib/kubelet/pods/a6526146-a37b-4fae-94ab-53e7c749c66d/volumes" Oct 30 13:25:29.701201 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:25:29.701162 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:25:37.338195 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:37.338150 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k5dtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:25:37.338908 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:37.338240 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5dtc" podUID="07364b75-84b5-43af-8e63-c447d9f49f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:25:37.409085 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:37.409044 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8ndvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:25:37.409424 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:37.409382 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8ndvt" podUID="d4d5e227-0a76-4cb2-b460-8409c61a9f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:25:52.417630 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:52.417090 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gdg2m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:25:52.417630 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:52.417181 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gdg2m" podUID="f6fc3d60-979c-4a16-9834-7c1083464b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:25:52.658479 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:52.658102 4503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x9h2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:25:52.658479 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:25:52.658165 4503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9h2r" podUID="dfc85b9a-8b4e-44fb-8de4-635cf391a3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:25:59.700690 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:25:59.700654 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:26:27.880591 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:27.880559 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bd48f45d7-9tch9_c1cd9a98-2a23-4361-a7c1-a12a456f8a69/manager/0.log" Oct 30 13:26:28.520281 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:28.520242 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-qlkgt_ec8777f0-af67-4906-8537-631e712dd6b6/manager/0.log" Oct 30 13:26:29.003878 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:29.003851 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25_34145300-0dc5-48e9-91ea-945034a3e538/copy-manifests/0.log" Oct 30 13:26:29.039549 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:29.039527 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25_34145300-0dc5-48e9-91ea-945034a3e538/manager/0.log" Oct 30 13:26:29.700754 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:26:29.700714 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:26:30.292737 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:30.292705 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-9g547_6ff77317-aae4-46c5-8e69-b5ede8f43cdf/maas-api/0.log" Oct 30 13:26:30.857919 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:30.857889 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-c4djd_844ba902-8fbc-4051-9872-32ec7a2fc96d/revoke-keys/0.log" Oct 30 13:26:31.044036 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:31.044005 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-d64vz_1267dbda-9b9c-4e67-842e-6b5a03c32650/revoke-keys/0.log" Oct 30 13:26:31.977404 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:31.977374 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-5r8dh_3cb49bf1-1cc0-4b74-b3cb-0e326ce064df/revoke-keys/0.log" Oct 30 13:26:32.336034 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:32.335927 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-qfpgx_bf42da6b-55a9-41e6-8881-da4580c09406/postgres/0.log" Oct 30 13:26:34.633458 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:34.633420 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-9s5j4_ae9ceeec-34bb-44cf-8319-c32da817aae4/manager/0.log" Oct 30 13:26:35.077326 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:35.077294 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-czj25_9b175e7a-d7b0-4425-9879-2d256d2f0324/manager/0.log" Oct 30 13:26:36.043109 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:36.043076 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wrf4q_dfe5fa31-1804-404f-8b06-3e3c2dd8e42e/discovery/0.log" Oct 30 13:26:36.241700 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:36.241673 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d59dd5dc7-8dvmm_da32ea91-80d7-48d5-9167-882b4736108e/kube-auth-proxy/0.log" Oct 30 13:26:36.840397 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:36.840365 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-8ndvt_d4d5e227-0a76-4cb2-b460-8409c61a9f12/istio-proxy/0.log" Oct 30 13:26:37.853668 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:37.853632 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-gdg2m_f6fc3d60-979c-4a16-9834-7c1083464b2c/istio-proxy/0.log" Oct 30 13:26:38.061734 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:38.061697 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-k5dtc_07364b75-84b5-43af-8e63-c447d9f49f3c/istio-proxy/0.log" Oct 30 13:26:38.481393 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:38.481369 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-x9h2r_dfc85b9a-8b4e-44fb-8de4-635cf391a3cd/istio-proxy/0.log" Oct 30 13:26:40.027004 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:40.026932 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf_a9e69714-f702-4f5b-9fdf-a1333076beb4/main/0.log" Oct 30 13:26:40.218630 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:40.218603 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-7tjtx_5522db7b-cfb1-4355-bbe0-b5022eac5e82/main/0.log" Oct 30 13:26:40.581222 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:40.581194 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-g9g7p_0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf/main/0.log" Oct 30 13:26:40.779764 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:40.779732 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr_76757b33-71a1-4186-8ab0-b24bfb08d1c8/main/0.log" Oct 30 13:26:40.967732 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:40.967704 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk_fd23c98d-5896-493c-a5f9-b02cd14c3869/main/0.log" Oct 30 13:26:48.054724 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:48.054689 4503 scope.go:122] "RemoveContainer" containerID="34cbd05f4deb8fed8fcf3929ba5bcebcbd0c41b10735de1d61179f3bc629a0db" Oct 30 13:26:51.867818 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:51.867785 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bd48f45d7-9tch9_c1cd9a98-2a23-4361-a7c1-a12a456f8a69/manager/0.log" Oct 30 13:26:52.532476 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:52.532444 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-qlkgt_ec8777f0-af67-4906-8537-631e712dd6b6/manager/0.log" Oct 30 13:26:52.986714 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:52.986685 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25_34145300-0dc5-48e9-91ea-945034a3e538/copy-manifests/0.log" Oct 30 13:26:53.017620 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:53.017593 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bb4ff5b5c-qrn25_34145300-0dc5-48e9-91ea-945034a3e538/manager/0.log" Oct 30 13:26:54.052817 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:54.052787 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-9g547_6ff77317-aae4-46c5-8e69-b5ede8f43cdf/maas-api/0.log" Oct 30 13:26:54.661431 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:54.661399 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-c4djd_844ba902-8fbc-4051-9872-32ec7a2fc96d/revoke-keys/0.log" Oct 30 13:26:54.872138 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:54.872111 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-d64vz_1267dbda-9b9c-4e67-842e-6b5a03c32650/revoke-keys/0.log" Oct 30 13:26:55.831816 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:55.831787 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-5r8dh_3cb49bf1-1cc0-4b74-b3cb-0e326ce064df/revoke-keys/0.log" Oct 30 13:26:56.241597 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:56.241566 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-qfpgx_bf42da6b-55a9-41e6-8881-da4580c09406/postgres/0.log" Oct 30 13:26:58.138241 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:58.138210 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-9s5j4_ae9ceeec-34bb-44cf-8319-c32da817aae4/manager/0.log" Oct 30 13:26:58.597855 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:58.597812 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-czj25_9b175e7a-d7b0-4425-9879-2d256d2f0324/manager/0.log" Oct 30 13:26:59.344471 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:59.344429 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wrf4q_dfe5fa31-1804-404f-8b06-3e3c2dd8e42e/discovery/0.log" Oct 30 13:26:59.520612 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:26:59.520577 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d59dd5dc7-8dvmm_da32ea91-80d7-48d5-9167-882b4736108e/kube-auth-proxy/0.log" Oct 30 13:26:59.700547 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:26:59.700509 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:27:00.106817 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:27:00.106731 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-8ndvt_d4d5e227-0a76-4cb2-b460-8409c61a9f12/istio-proxy/0.log" Oct 30 13:27:01.124327 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:27:01.124299 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-gdg2m_f6fc3d60-979c-4a16-9834-7c1083464b2c/istio-proxy/0.log" Oct 30 13:27:01.330984 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:27:01.330922 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-k5dtc_07364b75-84b5-43af-8e63-c447d9f49f3c/istio-proxy/0.log" Oct 30 13:27:01.743097 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:27:01.743066 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-x9h2r_dfc85b9a-8b4e-44fb-8de4-635cf391a3cd/istio-proxy/0.log" Oct 30 13:27:02.800861 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:27:02.800825 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-2kbcf_a9e69714-f702-4f5b-9fdf-a1333076beb4/main/0.log" Oct 30 13:27:02.987505 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:27:02.987478 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-7tjtx_5522db7b-cfb1-4355-bbe0-b5022eac5e82/main/0.log" Oct 30 13:27:03.371358 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:27:03.371331 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-g9g7p_0cf5cc67-e95c-4cf2-95dc-4ea9697fb4bf/main/0.log" Oct 30 13:27:03.548831 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:27:03.548794 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86pjrr_76757b33-71a1-4186-8ab0-b24bfb08d1c8/main/0.log" Oct 30 13:27:03.738790 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:27:03.738757 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-lzwzk_fd23c98d-5896-493c-a5f9-b02cd14c3869/main/0.log" Oct 30 13:27:05.037245 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:27:05.037203 4503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh"] Oct 30 13:27:05.058259 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:27:05.058221 4503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5r8dh"] Oct 30 13:27:06.360848 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:27:06.360815 4503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cb49bf1-1cc0-4b74-b3cb-0e326ce064df" path="/var/lib/kubelet/pods/3cb49bf1-1cc0-4b74-b3cb-0e326ce064df/volumes" Oct 30 13:27:29.701686 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:27:29.701639 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:27:39.780121 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:27:39.780087 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bd48f45d7-9tch9_c1cd9a98-2a23-4361-a7c1-a12a456f8a69/manager/0.log" Oct 30 13:27:40.080302 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:27:40.080209 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bd48f45d7-9tch9_c1cd9a98-2a23-4361-a7c1-a12a456f8a69/manager/0.log" Oct 30 13:27:59.701321 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:27:59.701285 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:28:23.843418 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:23.843359 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bd48f45d7-9tch9_c1cd9a98-2a23-4361-a7c1-a12a456f8a69/manager/0.log" Oct 30 13:28:24.118513 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:24.118409 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bd48f45d7-9tch9_c1cd9a98-2a23-4361-a7c1-a12a456f8a69/manager/0.log" Oct 30 13:28:29.700787 ip-10-0-130-200 kubenswrapper[4503]: E1030 13:28:29.700705 4503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hbsfb" containerName="extractor" Oct 30 13:28:30.119316 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:30.119233 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-r5lrn_87f27309-51ca-4116-b0e9-10cf9e144c36/global-pull-secret-syncer/0.log" Oct 30 13:28:31.089242 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:31.089214 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pz66k_51f1ecdc-2207-4ac7-9193-77f917cc0143/konnectivity-agent/0.log" Oct 30 13:28:31.359075 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:31.358980 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-200.ec2.internal_f1dd3ede18217038ba719ab8e2dbe78f/haproxy/0.log" Oct 30 13:28:38.301396 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:38.301355 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-9s5j4_ae9ceeec-34bb-44cf-8319-c32da817aae4/manager/0.log" Oct 30 13:28:38.574323 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:38.574217 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-czj25_9b175e7a-d7b0-4425-9879-2d256d2f0324/manager/0.log" Oct 30 13:28:39.115739 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.115701 4503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx"] Oct 30 13:28:39.116207 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.116189 4503 cpu_manager.go:414] "containerMap: removing container" podUID="a6526146-a37b-4fae-94ab-53e7c749c66d" containerName="maas-api" Oct 30 13:28:39.116256 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.116208 4503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6526146-a37b-4fae-94ab-53e7c749c66d" containerName="maas-api" Oct 30 13:28:39.116330 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.116321 4503 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6526146-a37b-4fae-94ab-53e7c749c66d" containerName="maas-api" Oct 30 13:28:39.119497 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.119480 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" Oct 30 13:28:39.137647 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.137618 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z6jh4\"/\"openshift-service-ca.crt\"" Oct 30 13:28:39.139542 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.139515 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx"] Oct 30 13:28:39.139894 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.139874 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e384fc26-73b7-4a6b-81b6-b4a433ee54a9-podres\") pod \"perf-node-gather-daemonset-rkgpx\" (UID: \"e384fc26-73b7-4a6b-81b6-b4a433ee54a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" Oct 30 13:28:39.139969 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.139910 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e384fc26-73b7-4a6b-81b6-b4a433ee54a9-sys\") pod \"perf-node-gather-daemonset-rkgpx\" (UID: \"e384fc26-73b7-4a6b-81b6-b4a433ee54a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" Oct 30 13:28:39.140115 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.140095 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e384fc26-73b7-4a6b-81b6-b4a433ee54a9-lib-modules\") pod \"perf-node-gather-daemonset-rkgpx\" (UID: \"e384fc26-73b7-4a6b-81b6-b4a433ee54a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" Oct 30 13:28:39.140181 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.140157 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z8bd\" (UniqueName: \"kubernetes.io/projected/e384fc26-73b7-4a6b-81b6-b4a433ee54a9-kube-api-access-5z8bd\") pod \"perf-node-gather-daemonset-rkgpx\" (UID: \"e384fc26-73b7-4a6b-81b6-b4a433ee54a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" Oct 30 13:28:39.140236 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.140184 4503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e384fc26-73b7-4a6b-81b6-b4a433ee54a9-proc\") pod \"perf-node-gather-daemonset-rkgpx\" (UID: \"e384fc26-73b7-4a6b-81b6-b4a433ee54a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" Oct 30 13:28:39.149056 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.149033 4503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z6jh4\"/\"kube-root-ca.crt\"" Oct 30 13:28:39.149154 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.149108 4503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z6jh4\"/\"default-dockercfg-hfvsn\"" Oct 30 13:28:39.241583 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.241545 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e384fc26-73b7-4a6b-81b6-b4a433ee54a9-podres\") pod \"perf-node-gather-daemonset-rkgpx\" (UID: \"e384fc26-73b7-4a6b-81b6-b4a433ee54a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" Oct 30 13:28:39.241791 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.241597 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e384fc26-73b7-4a6b-81b6-b4a433ee54a9-sys\") pod \"perf-node-gather-daemonset-rkgpx\" (UID: \"e384fc26-73b7-4a6b-81b6-b4a433ee54a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" Oct 30 13:28:39.241791 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.241672 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e384fc26-73b7-4a6b-81b6-b4a433ee54a9-lib-modules\") pod \"perf-node-gather-daemonset-rkgpx\" (UID: \"e384fc26-73b7-4a6b-81b6-b4a433ee54a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" Oct 30 13:28:39.241791 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.241678 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e384fc26-73b7-4a6b-81b6-b4a433ee54a9-podres\") pod \"perf-node-gather-daemonset-rkgpx\" (UID: \"e384fc26-73b7-4a6b-81b6-b4a433ee54a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" Oct 30 13:28:39.241791 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.241716 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z8bd\" (UniqueName: \"kubernetes.io/projected/e384fc26-73b7-4a6b-81b6-b4a433ee54a9-kube-api-access-5z8bd\") pod \"perf-node-gather-daemonset-rkgpx\" (UID: \"e384fc26-73b7-4a6b-81b6-b4a433ee54a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" Oct 30 13:28:39.241791 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.241728 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e384fc26-73b7-4a6b-81b6-b4a433ee54a9-sys\") pod \"perf-node-gather-daemonset-rkgpx\" (UID: \"e384fc26-73b7-4a6b-81b6-b4a433ee54a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" Oct 30 13:28:39.241791 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.241745 4503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e384fc26-73b7-4a6b-81b6-b4a433ee54a9-proc\") pod \"perf-node-gather-daemonset-rkgpx\" (UID: \"e384fc26-73b7-4a6b-81b6-b4a433ee54a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" Oct 30 13:28:39.241791 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.241757 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e384fc26-73b7-4a6b-81b6-b4a433ee54a9-lib-modules\") pod \"perf-node-gather-daemonset-rkgpx\" (UID: \"e384fc26-73b7-4a6b-81b6-b4a433ee54a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" Oct 30 13:28:39.242091 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.241814 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e384fc26-73b7-4a6b-81b6-b4a433ee54a9-proc\") pod \"perf-node-gather-daemonset-rkgpx\" (UID: \"e384fc26-73b7-4a6b-81b6-b4a433ee54a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" Oct 30 13:28:39.266635 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.266605 4503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z8bd\" (UniqueName: \"kubernetes.io/projected/e384fc26-73b7-4a6b-81b6-b4a433ee54a9-kube-api-access-5z8bd\") pod \"perf-node-gather-daemonset-rkgpx\" (UID: \"e384fc26-73b7-4a6b-81b6-b4a433ee54a9\") " pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" Oct 30 13:28:39.433290 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.433256 4503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" Oct 30 13:28:39.732963 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.732921 4503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx"] Oct 30 13:28:39.733243 ip-10-0-130-200 kubenswrapper[4503]: W1030 13:28:39.733212 4503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode384fc26_73b7_4a6b_81b6_b4a433ee54a9.slice/crio-f8ea0a5ccd2868256011f8181034755835f382197f23fc73376a854a76189c4c WatchSource:0}: Error finding container f8ea0a5ccd2868256011f8181034755835f382197f23fc73376a854a76189c4c: Status 404 returned error can't find the container with id f8ea0a5ccd2868256011f8181034755835f382197f23fc73376a854a76189c4c Oct 30 13:28:39.974787 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:39.974758 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15352ec1-7ce6-4cfe-8427-02727875d683/alertmanager/0.log" Oct 30 13:28:40.090432 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:40.090363 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15352ec1-7ce6-4cfe-8427-02727875d683/config-reloader/0.log" Oct 30 13:28:40.205828 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:40.205795 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15352ec1-7ce6-4cfe-8427-02727875d683/kube-rbac-proxy-web/0.log" Oct 30 13:28:40.318857 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:40.318827 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15352ec1-7ce6-4cfe-8427-02727875d683/kube-rbac-proxy/0.log" Oct 30 13:28:40.440528 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:40.440501 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15352ec1-7ce6-4cfe-8427-02727875d683/kube-rbac-proxy-metric/0.log" Oct 30 13:28:40.594364 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:40.594330 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" event={"ID":"e384fc26-73b7-4a6b-81b6-b4a433ee54a9","Type":"ContainerStarted","Data":"a870a15f6c31f867102f92a3bf5abdefe6be1c14547ce3b87fc1b32bd968c85c"} Oct 30 13:28:40.594563 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:40.594378 4503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" event={"ID":"e384fc26-73b7-4a6b-81b6-b4a433ee54a9","Type":"ContainerStarted","Data":"f8ea0a5ccd2868256011f8181034755835f382197f23fc73376a854a76189c4c"} Oct 30 13:28:40.594563 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:40.594468 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" Oct 30 13:28:40.596401 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:40.596382 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15352ec1-7ce6-4cfe-8427-02727875d683/prom-label-proxy/0.log" Oct 30 13:28:40.640270 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:40.640219 4503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" podStartSLOduration=1.6402044409999998 podStartE2EDuration="1.640204441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:40.638062939 +0000 UTC m=+2214.461291216" watchObservedRunningTime="2026-10-30 13:28:40.640204441 +0000 UTC m=+2214.463432718" Oct 30 13:28:40.714646 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:40.714532 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_15352ec1-7ce6-4cfe-8427-02727875d683/init-config-reloader/0.log" Oct 30 13:28:41.920426 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:41.920398 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dlsxw_933dffcb-401a-42ad-8233-d1cf1ce231e3/node-exporter/0.log" Oct 30 13:28:42.029582 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:42.029557 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dlsxw_933dffcb-401a-42ad-8233-d1cf1ce231e3/kube-rbac-proxy/0.log" Oct 30 13:28:42.138590 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:42.138565 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dlsxw_933dffcb-401a-42ad-8233-d1cf1ce231e3/init-textfile/0.log" Oct 30 13:28:42.597760 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:42.597725 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6nlzd_3cd66bfb-6bb5-46e1-b0ae-91f0520222ed/kube-rbac-proxy-main/0.log" Oct 30 13:28:42.709022 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:42.708965 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6nlzd_3cd66bfb-6bb5-46e1-b0ae-91f0520222ed/kube-rbac-proxy-self/0.log" Oct 30 13:28:42.821549 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:42.821519 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6nlzd_3cd66bfb-6bb5-46e1-b0ae-91f0520222ed/openshift-state-metrics/0.log" Oct 30 13:28:43.988108 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:43.988063 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-7jdw7_3df4074f-8df3-47a4-8aa8-7618640bc487/prometheus-operator-admission-webhook/0.log" Oct 30 13:28:44.456752 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:44.456723 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696fdb48d6-54hj7_9bd07a93-063c-4c0e-bff6-afd530af1fe2/thanos-query/0.log" Oct 30 13:28:44.567222 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:44.567158 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696fdb48d6-54hj7_9bd07a93-063c-4c0e-bff6-afd530af1fe2/kube-rbac-proxy-web/0.log" Oct 30 13:28:44.680201 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:44.680164 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696fdb48d6-54hj7_9bd07a93-063c-4c0e-bff6-afd530af1fe2/kube-rbac-proxy/0.log" Oct 30 13:28:44.787222 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:44.787151 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696fdb48d6-54hj7_9bd07a93-063c-4c0e-bff6-afd530af1fe2/prom-label-proxy/0.log" Oct 30 13:28:44.897567 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:44.897537 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696fdb48d6-54hj7_9bd07a93-063c-4c0e-bff6-afd530af1fe2/kube-rbac-proxy-rules/0.log" Oct 30 13:28:45.010567 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:45.010539 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696fdb48d6-54hj7_9bd07a93-063c-4c0e-bff6-afd530af1fe2/kube-rbac-proxy-metrics/0.log" Oct 30 13:28:46.610184 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:46.610157 4503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-z6jh4/perf-node-gather-daemonset-rkgpx" Oct 30 13:28:48.089156 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:48.089119 4503 scope.go:122] "RemoveContainer" containerID="da86d0df9963ddb545781eda70474a1309094ede557b98ab0f45ad57eff0b290" Oct 30 13:28:48.608679 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:48.608636 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-g7dlx_e69c2260-dd1b-4a31-b08b-0e6681fa76c1/download-server/0.log" Oct 30 13:28:51.906702 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:51.906668 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b7kx7_6e7c1377-fb2f-4b5f-83a7-b799206fb344/dns/0.log" Oct 30 13:28:52.045335 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:52.045298 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b7kx7_6e7c1377-fb2f-4b5f-83a7-b799206fb344/kube-rbac-proxy/0.log" Oct 30 13:28:52.913776 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:52.913742 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z6m2l_c10fc6a1-af69-48c2-b3dc-131e48f36b22/dns-node-resolver/0.log" Oct 30 13:28:53.722186 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:53.722142 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wrf4q_dfe5fa31-1804-404f-8b06-3e3c2dd8e42e/discovery/0.log" Oct 30 13:28:53.756282 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:53.756251 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d59dd5dc7-8dvmm_da32ea91-80d7-48d5-9167-882b4736108e/kube-auth-proxy/0.log" Oct 30 13:28:53.934830 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:53.934802 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-57c578b6f7-d78ps_3a8ee171-fa17-418f-a59b-2cc6bd5aa2fe/registry/0.log" Oct 30 13:28:53.962980 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:53.962934 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-8ndvt_d4d5e227-0a76-4cb2-b460-8409c61a9f12/istio-proxy/0.log" Oct 30 13:28:54.168743 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:54.168711 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kvsqc_10b4d740-fb6d-486e-b6c1-5c6115a10496/node-ca/0.log" Oct 30 13:28:54.302005 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:54.301969 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-gdg2m_f6fc3d60-979c-4a16-9834-7c1083464b2c/istio-proxy/0.log" Oct 30 13:28:54.368099 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:54.368076 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-k5dtc_07364b75-84b5-43af-8e63-c447d9f49f3c/istio-proxy/0.log" Oct 30 13:28:54.479458 ip-10-0-130-200 kubenswrapper[4503]: I1030 13:28:54.479383 4503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-x9h2r_dfc85b9a-8b4e-44fb-8de4-635cf391a3cd/istio-proxy/0.log"