Nov 04 03:46:23.273136 ip-10-0-138-9 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 04 03:46:23.273148 ip-10-0-138-9 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 04 03:46:23.273158 ip-10-0-138-9 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 04 03:46:23.273507 ip-10-0-138-9 systemd[1]: Failed to start Kubernetes Kubelet. Nov 04 03:46:33.381187 ip-10-0-138-9 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 04 03:46:33.381205 ip-10-0-138-9 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 0ca5310f37da482c88d9e2016640dff1 -- Nov 04 03:48:33.429885 ip-10-0-138-9 systemd[1]: Starting Kubernetes Kubelet... Nov 04 03:48:33.813413 ip-10-0-138-9 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 03:48:33.813413 ip-10-0-138-9 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 03:48:33.813413 ip-10-0-138-9 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 03:48:33.813413 ip-10-0-138-9 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 03:48:33.819614 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819524 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:48:33.819703 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819622 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:48:33.819703 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819629 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:48:33.819703 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819632 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:48:33.819703 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819635 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:48:33.819703 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819638 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:48:33.819703 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819641 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:48:33.819703 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819644 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:48:33.819703 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819648 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:48:33.819703 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819651 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:48:33.819703 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819653 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:48:33.819703 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819656 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:48:33.819703 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819658 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:48:33.819703 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819662 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:48:33.819703 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819664 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:48:33.819703 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819667 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:48:33.819703 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819670 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:48:33.819703 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819672 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:48:33.819703 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819675 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:48:33.819703 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819677 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:48:33.820163 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819680 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:48:33.820163 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819682 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:48:33.820163 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819685 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:48:33.820163 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819688 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:48:33.820163 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819692 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:48:33.820163 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819694 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:48:33.820163 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819697 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:48:33.820163 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819699 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:48:33.820163 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819702 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:48:33.820163 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819705 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:48:33.820163 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819707 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:48:33.820163 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819710 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:48:33.820163 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819713 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:48:33.820163 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819716 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:48:33.820163 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819719 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:48:33.820163 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819722 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:48:33.820163 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819724 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:48:33.820163 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819727 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:48:33.820163 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819729 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:48:33.820623 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819733 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:48:33.820623 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819739 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:48:33.820623 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819742 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:48:33.820623 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819745 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:48:33.820623 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819747 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:48:33.820623 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819750 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:48:33.820623 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819752 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:48:33.820623 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819755 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:48:33.820623 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819757 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:48:33.820623 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819760 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:48:33.820623 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819762 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:48:33.820623 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819765 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:48:33.820623 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819768 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:48:33.820623 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819771 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:48:33.820623 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819774 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:48:33.820623 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819777 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:48:33.820623 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819779 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:48:33.820623 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819782 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:48:33.820623 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819785 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:48:33.821071 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819787 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:48:33.821071 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819790 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:48:33.821071 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819793 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:48:33.821071 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819795 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:48:33.821071 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819798 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:48:33.821071 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819801 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:48:33.821071 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819803 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:48:33.821071 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819805 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:48:33.821071 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819809 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:48:33.821071 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819812 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:48:33.821071 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819814 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:48:33.821071 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819817 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:48:33.821071 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819819 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:48:33.821071 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819823 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:48:33.821071 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819826 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:48:33.821071 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819829 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:48:33.821071 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819832 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:48:33.821071 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819835 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:48:33.821071 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819838 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819841 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819844 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819846 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819849 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819852 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819854 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819858 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819861 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819864 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819867 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819869 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819872 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819875 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819877 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819880 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819883 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819885 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819889 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819891 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:48:33.821526 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819894 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:48:33.822008 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819896 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:48:33.822008 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819899 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:48:33.822008 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819901 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:48:33.822008 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819904 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:48:33.822008 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819906 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:48:33.822008 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819909 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:48:33.822008 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819912 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:48:33.822008 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819915 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:48:33.822008 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.819918 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:48:33.822008 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820698 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:48:33.822008 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820706 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:48:33.822008 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820709 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:48:33.822008 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820712 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:48:33.822008 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820715 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:48:33.822008 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820717 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:48:33.822008 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820720 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:48:33.822008 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820722 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:48:33.822008 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820725 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:48:33.822008 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820728 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:48:33.822462 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820731 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:48:33.822462 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820734 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:48:33.822462 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820738 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:48:33.822462 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820742 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:48:33.822462 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820745 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:48:33.822462 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820748 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:48:33.822462 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820750 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:48:33.822462 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820753 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:48:33.822462 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820755 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:48:33.822462 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820758 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:48:33.822462 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820760 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:48:33.822462 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820763 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:48:33.822462 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820767 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:48:33.822462 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820770 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:48:33.822462 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820773 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:48:33.822462 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820776 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:48:33.822462 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820779 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:48:33.822462 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820781 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:48:33.822462 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820784 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820787 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820790 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820793 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820796 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820798 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820801 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820804 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820806 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820809 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820812 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820814 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820817 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820819 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820822 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820825 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820828 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820830 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820833 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820836 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:48:33.822939 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820839 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820841 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820844 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820847 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820849 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820852 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820855 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820857 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820860 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820863 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820865 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820868 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820870 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820874 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820876 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820879 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820882 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820884 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820887 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820890 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:48:33.823425 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820893 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:48:33.823913 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820895 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:48:33.823913 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820898 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:48:33.823913 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820900 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:48:33.823913 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820903 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:48:33.823913 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820905 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:48:33.823913 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820907 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:48:33.823913 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820910 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:48:33.823913 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820913 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:48:33.823913 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820916 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:48:33.823913 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820918 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:48:33.823913 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820920 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:48:33.823913 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820923 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:48:33.823913 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820928 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:48:33.823913 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820931 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:48:33.823913 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820933 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:48:33.823913 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820936 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:48:33.823913 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820939 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:48:33.823913 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820941 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:48:33.823913 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820944 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:48:33.824360 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820946 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:48:33.824360 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820949 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:48:33.824360 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820951 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:48:33.824360 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820954 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:48:33.824360 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820956 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:48:33.824360 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820959 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:48:33.824360 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820962 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:48:33.824360 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820964 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:48:33.824360 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820967 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:48:33.824360 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820969 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:48:33.824360 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820972 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:48:33.824360 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820975 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:48:33.824360 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820977 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:48:33.824360 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820980 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:48:33.824360 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820983 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:48:33.824360 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820985 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:48:33.824360 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820989 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:48:33.824360 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.820992 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:48:33.824360 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821480 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821485 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821488 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821491 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821495 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821497 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821500 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821503 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821506 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821509 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821511 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821514 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821516 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821519 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821522 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821524 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821527 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821530 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821532 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821536 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:48:33.824815 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821539 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:48:33.825282 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821541 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:48:33.825282 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821544 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:48:33.825282 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821546 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:48:33.825282 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821549 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:48:33.825282 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821551 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:48:33.825282 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821555 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:48:33.825282 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821559 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:48:33.825282 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821561 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:48:33.825282 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821564 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:48:33.825282 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821567 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:48:33.825282 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821569 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:48:33.825282 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821572 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:48:33.825282 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821574 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:48:33.825282 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821577 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:48:33.825282 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821583 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:48:33.825282 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821586 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:48:33.825282 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821588 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:48:33.825282 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821591 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:48:33.825282 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821593 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821596 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821599 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821614 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821617 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821621 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821624 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821627 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821629 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821632 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821635 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821638 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821641 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821643 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821646 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821649 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821651 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821654 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821656 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821659 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:48:33.825760 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821661 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821664 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821667 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821669 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821672 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821674 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821677 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821680 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821682 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821686 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821688 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821691 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821693 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821696 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821698 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821701 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821704 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821706 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821709 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821712 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:48:33.826229 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821714 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821718 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821722 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821725 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821728 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821731 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821733 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821736 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821739 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821741 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821744 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821746 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821749 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821751 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821754 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821756 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821759 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821761 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821764 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821767 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:48:33.826749 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821769 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821771 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821780 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821783 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821786 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821788 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.821791 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.822918 2584 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.822933 2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.822939 2584 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.822944 2584 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.822949 2584 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.822952 2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.822957 2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.822961 2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.822965 2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.822968 2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.822972 2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.822975 2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.822978 2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.822981 2584 flags.go:64] FLAG: --cgroup-root="" Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.822984 2584 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 03:48:33.827219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.822988 2584 flags.go:64] FLAG: --client-ca-file="" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.822991 2584 flags.go:64] FLAG: --cloud-provider="external" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.822994 2584 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.822999 2584 flags.go:64] FLAG: --cluster-domain="" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823002 2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823008 2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823011 2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823014 2584 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823018 2584 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823021 2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823024 2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823028 2584 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823031 2584 flags.go:64] FLAG: --contention-profiling="false" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823035 2584 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823038 2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823041 2584 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823045 2584 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823050 2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823053 2584 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823056 2584 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823059 2584 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823062 2584 flags.go:64] FLAG: --enable-server="true" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823065 2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823072 2584 flags.go:64] FLAG: --event-burst="100" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823076 2584 flags.go:64] FLAG: --event-qps="50" Nov 04 03:48:33.827741 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823078 2584 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823082 2584 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823086 2584 flags.go:64] FLAG: --eviction-hard="" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823090 2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823093 2584 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823096 2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823099 2584 flags.go:64] FLAG: --eviction-soft="" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823102 2584 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823105 2584 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823108 2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823111 2584 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823114 2584 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823119 2584 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823122 2584 flags.go:64] FLAG: --feature-gates="" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823126 2584 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823129 2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823132 2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823135 2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823138 2584 flags.go:64] FLAG: --healthz-port="10248" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823141 2584 flags.go:64] FLAG: --help="false" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823144 2584 flags.go:64] FLAG: --hostname-override="ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823149 2584 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823152 2584 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 03:48:33.828382 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823155 2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823158 2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823162 2584 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823164 2584 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823167 2584 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823170 2584 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823173 2584 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823176 2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823179 2584 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823182 2584 flags.go:64] FLAG: --kube-reserved="" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823185 2584 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823188 2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823191 2584 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823194 2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823197 2584 flags.go:64] FLAG: --lock-file="" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823200 2584 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823203 2584 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823206 2584 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823211 2584 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823214 2584 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823217 2584 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823222 2584 flags.go:64] FLAG: --logging-format="text" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823224 2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823228 2584 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 03:48:33.828931 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823231 2584 flags.go:64] FLAG: --manifest-url="" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823233 2584 flags.go:64] FLAG: --manifest-url-header="" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823238 2584 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823241 2584 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823245 2584 flags.go:64] FLAG: --max-pods="110" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823248 2584 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823251 2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823255 2584 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823257 2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823261 2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823263 2584 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823266 2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823273 2584 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823276 2584 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823279 2584 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823282 2584 flags.go:64] FLAG: --pod-cidr="" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823285 2584 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823288 2584 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823292 2584 flags.go:64] FLAG: --pods-per-core="0" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823295 2584 flags.go:64] FLAG: --port="10250" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823298 2584 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823300 2584 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-08b3288f5328a15ef" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823304 2584 flags.go:64] FLAG: --qos-reserved="" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823306 2584 flags.go:64] FLAG: --read-only-port="10255" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823310 2584 flags.go:64] FLAG: --register-node="true" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823313 2584 flags.go:64] FLAG: --register-with-taints="" Nov 04 03:48:33.829526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823316 2584 flags.go:64] FLAG: --registry-burst="10" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823319 2584 flags.go:64] FLAG: --registry-qps="5" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823322 2584 flags.go:64] FLAG: --reserved-cpus="" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823326 2584 flags.go:64] FLAG: --reserved-memory="" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823330 2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823333 2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823336 2584 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823339 2584 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823342 2584 flags.go:64] FLAG: --runonce="false" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823345 2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823348 2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823351 2584 flags.go:64] FLAG: --seccomp-default="false" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823354 2584 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823358 2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823362 2584 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823365 2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823368 2584 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823371 2584 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823373 2584 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823376 2584 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823379 2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823382 2584 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823385 2584 flags.go:64] FLAG: --system-cgroups="" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823388 2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823394 2584 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 03:48:33.830643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823397 2584 flags.go:64] FLAG: --tls-cert-file="" Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823399 2584 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823404 2584 flags.go:64] FLAG: --tls-min-version="" Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823407 2584 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823410 2584 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823413 2584 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823416 2584 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823419 2584 flags.go:64] FLAG: --v="2" Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823423 2584 flags.go:64] FLAG: --version="false" Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823427 2584 flags.go:64] FLAG: --vmodule="" Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823433 2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823436 2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823574 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823578 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823580 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823585 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823588 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823591 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823594 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823596 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823599 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823616 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:48:33.831862 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823621 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:48:33.832441 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823625 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:48:33.832441 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823628 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:48:33.832441 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823631 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:48:33.832441 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823634 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:48:33.832441 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823636 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:48:33.832441 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823640 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:48:33.832441 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823642 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:48:33.832441 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823645 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:48:33.832441 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823647 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:48:33.832441 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823650 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:48:33.832441 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823652 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:48:33.832441 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823655 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:48:33.832441 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823657 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:48:33.832441 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823660 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:48:33.832441 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823663 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:48:33.832441 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823665 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:48:33.832441 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823668 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:48:33.832441 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823670 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:48:33.832441 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823673 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823677 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823680 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823682 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823685 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823687 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823690 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823692 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823698 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823701 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823703 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823706 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823709 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823711 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823715 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823717 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823722 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823725 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823728 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823731 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:48:33.833063 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823734 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:48:33.833951 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823736 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:48:33.833951 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823739 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:48:33.833951 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823741 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:48:33.833951 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823744 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:48:33.833951 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823746 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:48:33.833951 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823749 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:48:33.833951 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823752 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:48:33.833951 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823754 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:48:33.833951 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823756 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:48:33.833951 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823759 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:48:33.833951 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823762 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:48:33.833951 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823764 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:48:33.833951 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823767 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:48:33.833951 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823770 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:48:33.833951 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823773 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:48:33.833951 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823775 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:48:33.833951 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823778 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:48:33.833951 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823780 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:48:33.833951 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823783 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823787 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823790 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823792 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823795 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823797 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823800 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823803 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823806 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823809 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823811 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823814 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823816 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823819 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823821 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823824 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823826 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823829 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823831 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823834 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:48:33.834803 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823837 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:48:33.835338 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823839 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:48:33.835338 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823842 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:48:33.835338 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823844 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:48:33.835338 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823847 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:48:33.835338 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823849 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:48:33.835338 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823854 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:48:33.835338 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823857 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:48:33.835338 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823860 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:48:33.835338 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823862 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:48:33.835338 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823865 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:48:33.835338 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823868 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:48:33.835338 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823870 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:48:33.835338 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823873 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:48:33.835338 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823876 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:48:33.835338 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823879 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:48:33.835338 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823882 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:48:33.835338 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823884 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:48:33.835856 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.823911 2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 03:48:33.835856 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823949 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:48:33.835856 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823954 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:48:33.835856 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823961 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:48:33.835856 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823964 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:48:33.835856 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823967 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:48:33.835856 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823969 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:48:33.835856 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823972 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:48:33.835856 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823975 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:48:33.835856 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823977 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:48:33.835856 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823980 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:48:33.835856 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823983 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:48:33.835856 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823985 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:48:33.835856 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823988 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:48:33.835856 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823990 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:48:33.835856 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823993 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823995 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.823998 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824001 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824003 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824006 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824009 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824012 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824015 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824017 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824020 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824023 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824025 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824029 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824031 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824034 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824036 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824039 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824041 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824044 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:48:33.836264 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824047 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:48:33.837088 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824050 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:48:33.837088 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824053 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:48:33.837088 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824055 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:48:33.837088 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824058 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:48:33.837088 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824060 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:48:33.837088 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824063 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:48:33.837088 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824066 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:48:33.837088 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824068 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:48:33.837088 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824071 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:48:33.837088 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824074 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:48:33.837088 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824076 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:48:33.837088 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824079 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:48:33.837088 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824081 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:48:33.837088 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824084 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:48:33.837088 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824086 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:48:33.837088 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824089 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:48:33.837088 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824092 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:48:33.837088 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824094 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:48:33.837088 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824097 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:48:33.837891 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824100 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:48:33.837891 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824104 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:48:33.837891 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824107 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:48:33.837891 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824109 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:48:33.837891 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824112 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:48:33.837891 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824116 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:48:33.837891 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824118 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:48:33.837891 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824121 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:48:33.837891 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824124 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:48:33.837891 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824127 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:48:33.837891 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824129 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:48:33.837891 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824132 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:48:33.837891 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824134 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:48:33.837891 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824138 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:48:33.837891 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824141 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:48:33.837891 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824143 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:48:33.837891 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824146 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:48:33.837891 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824148 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:48:33.837891 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824151 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824153 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824156 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824158 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824161 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824163 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824166 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824169 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824171 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824174 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824176 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824179 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824181 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824184 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824186 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824189 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824191 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824194 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824196 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824200 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:48:33.838394 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824203 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:48:33.838969 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824205 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:48:33.838969 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824208 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:48:33.838969 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824210 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:48:33.838969 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824214 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:48:33.838969 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824217 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:48:33.838969 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824220 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:48:33.838969 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824223 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:48:33.838969 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824226 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:48:33.838969 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824229 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:48:33.838969 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824232 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:48:33.838969 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824235 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:48:33.838969 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824238 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:48:33.838969 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.824240 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:48:33.838969 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.831928 2584 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 03:48:33.838969 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.831950 2584 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 03:48:33.838969 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.832809 2584 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 03:48:33.838969 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.832821 2584 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 03:48:33.838969 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.832912 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:48:33.838969 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.832920 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:48:33.839467 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.832928 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:48:33.839467 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.832934 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:48:33.839467 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.832940 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:48:33.839467 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.832944 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:48:33.839467 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.832949 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:48:33.839467 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.832954 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:48:33.839467 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.832958 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:48:33.839467 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.832963 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:48:33.839467 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.832967 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:48:33.839467 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.832971 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:48:33.839467 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.832975 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:48:33.839467 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.832980 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:48:33.839467 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.832984 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:48:33.839467 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.832989 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:48:33.839467 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.832994 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:48:33.839467 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.832998 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:48:33.839467 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833002 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:48:33.839467 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833006 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:48:33.839467 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833011 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:48:33.839973 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833015 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:48:33.839973 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833019 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:48:33.839973 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833024 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:48:33.839973 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833028 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:48:33.839973 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833033 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:48:33.839973 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833037 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:48:33.839973 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833042 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:48:33.839973 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833046 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:48:33.839973 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833051 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:48:33.839973 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833056 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:48:33.839973 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833060 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:48:33.839973 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833064 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:48:33.839973 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833068 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:48:33.839973 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833072 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:48:33.839973 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833076 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:48:33.839973 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833081 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:48:33.839973 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833085 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:48:33.839973 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833089 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:48:33.839973 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833094 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833098 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833102 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833107 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833111 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833116 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833120 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833125 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833129 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833133 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833138 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833142 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833147 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833151 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833155 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833160 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833165 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833174 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833178 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833183 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:48:33.840476 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833187 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833192 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833196 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833202 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833206 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833211 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833215 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833219 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833224 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833228 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833232 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833236 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833241 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833245 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833249 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833253 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833257 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833261 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833265 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833270 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:48:33.841059 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833274 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:48:33.841907 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833278 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:48:33.841907 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833283 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:48:33.841907 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833287 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:48:33.841907 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833292 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:48:33.841907 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833296 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:48:33.841907 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833301 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:48:33.841907 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833305 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:48:33.841907 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833310 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:48:33.841907 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833314 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:48:33.841907 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833319 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:48:33.841907 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833323 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:48:33.841907 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833327 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:48:33.841907 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833331 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:48:33.841907 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833336 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:48:33.841907 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833340 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:48:33.841907 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833346 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:48:33.841907 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833351 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:48:33.841907 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833356 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:48:33.841907 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833360 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:48:33.842687 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833364 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:48:33.842687 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833371 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:48:33.842687 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833378 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:48:33.842687 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833383 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:48:33.842687 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833387 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:48:33.842687 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833392 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:48:33.842687 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833396 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:48:33.842687 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.833439 2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 03:48:33.842687 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833679 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:48:33.842687 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833687 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:48:33.842687 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833692 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:48:33.842687 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833697 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:48:33.842687 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833702 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:48:33.842687 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833706 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:48:33.842687 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833710 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:48:33.842687 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833714 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833718 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833722 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833726 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833730 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833735 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833739 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833744 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833748 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833753 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833758 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833762 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833766 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833770 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833775 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833779 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833784 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833788 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833793 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833797 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:48:33.843076 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833801 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833805 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833809 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833813 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833818 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833821 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833826 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833830 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833835 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833839 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833843 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833847 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833851 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833855 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833859 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833863 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833867 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833871 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833876 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833886 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:48:33.843567 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833890 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833895 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833899 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833904 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833908 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833912 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833916 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833922 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833926 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833930 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833934 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833939 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833943 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833947 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833951 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833955 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833958 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833962 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833966 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833970 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:48:33.844053 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833974 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833978 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833983 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833987 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833992 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.833997 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834000 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834004 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834008 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834013 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834017 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834022 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834026 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834030 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834035 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834039 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834044 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834048 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834052 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834056 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:48:33.844572 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834063 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:48:33.845078 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834070 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:48:33.845078 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834075 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:48:33.845078 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834079 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:48:33.845078 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834083 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:48:33.845078 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834088 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:48:33.845078 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834092 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:48:33.845078 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834097 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:48:33.845078 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834101 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:48:33.845078 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834105 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:48:33.845078 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834111 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:48:33.845078 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834118 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:48:33.845078 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834123 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:48:33.845078 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834128 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:48:33.845078 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834133 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:48:33.845078 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834137 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:48:33.845078 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834141 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:48:33.845078 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834146 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:48:33.845078 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834150 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:48:33.845504 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.834191 2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 03:48:33.845504 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834239 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:48:33.845504 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834248 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:48:33.845504 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834254 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:48:33.845504 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834258 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:48:33.845504 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834263 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:48:33.845504 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834267 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:48:33.845504 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834271 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:48:33.845504 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834275 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:48:33.845504 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834279 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:48:33.845504 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834284 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:48:33.845504 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834288 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:48:33.845504 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834293 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:48:33.845504 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834297 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:48:33.845504 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834302 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:48:33.845504 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834306 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834311 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834315 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834319 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834323 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834328 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834332 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834337 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834341 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834345 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834349 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834354 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834358 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834362 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834366 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834371 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834375 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834380 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834384 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834388 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:48:33.846149 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834392 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834396 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834400 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834404 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834408 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834413 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834417 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834421 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834426 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834431 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834435 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834439 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834444 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834449 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834453 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834457 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834461 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834465 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834474 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834478 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:48:33.846921 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834482 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:48:33.847697 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834487 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:48:33.847697 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834491 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:48:33.847697 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834497 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:48:33.847697 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834504 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:48:33.847697 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834509 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:48:33.847697 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834514 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:48:33.847697 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834520 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:48:33.847697 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834525 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:48:33.847697 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834529 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:48:33.847697 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834533 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:48:33.847697 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834538 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:48:33.847697 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834542 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:48:33.847697 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834549 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:48:33.847697 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834555 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:48:33.847697 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834561 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:48:33.847697 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834566 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:48:33.847697 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834570 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:48:33.847697 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834574 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834578 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834583 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834587 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834592 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834596 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834617 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834623 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834627 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834631 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834635 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834639 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834645 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834649 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834653 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834657 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834662 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834666 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834671 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834675 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:48:33.848110 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834679 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:48:33.848624 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834688 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:48:33.848624 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834692 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:48:33.848624 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834696 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:48:33.848624 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834701 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:48:33.848624 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834705 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:48:33.848624 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834709 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:48:33.848624 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834713 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:48:33.848624 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834718 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:48:33.848624 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834723 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:48:33.848624 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834727 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:48:33.848624 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834731 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:48:33.848624 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834735 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:48:33.848624 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:33.834740 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:48:33.848624 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.834813 2584 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 03:48:33.848624 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.838358 2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 04 03:48:33.848624 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.839336 2584 server.go:1014] "Starting client certificate rotation" Nov 04 03:48:33.848624 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.839435 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 03:48:33.848624 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.839469 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 03:48:33.849044 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.841512 2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 03:48:33.849044 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.848090 2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 03:48:33.857525 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.857508 2584 log.go:25] "Validated CRI v1 runtime API" Nov 04 03:48:33.861352 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.861336 2584 log.go:25] "Validated CRI v1 image API" Nov 04 03:48:33.862221 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.862192 2584 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 03:48:33.865138 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.865111 2584 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9cf85cde-b97e-42df-94bc-7570fe38d0f5:/dev/nvme0n1p3 bfe504cc-bf8b-4943-b41e-9f868b7133ca:/dev/nvme0n1p4] Nov 04 03:48:33.865210 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.865136 2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 04 03:48:33.870024 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.869903 2584 manager.go:217] Machine: {Timestamp:2026-11-04 03:48:33.868743237 +0000 UTC m=+0.324968025 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103406 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec219d89ae6f36ba96bceaa767421753 SystemUUID:ec219d89-ae6f-36ba-96bc-eaa767421753 BootID:0ca5310f-37da-482c-88d9-e2016640dff1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f9:01:2f:19 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f9:01:2f:19 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:4a:65:c3:5b:9d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 03:48:33.870024 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.870019 2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 03:48:33.870175 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.870158 2584 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 03:48:33.872324 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.872309 2584 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 03:48:33.873257 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.873233 2584 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 03:48:33.873391 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.873260 2584 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-9.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 03:48:33.874012 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.874001 2584 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 03:48:33.874043 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.874014 2584 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 03:48:33.874070 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.874041 2584 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 03:48:33.874070 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.874057 2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 03:48:33.874743 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.874732 2584 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 03:48:33.878340 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.878327 2584 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 03:48:33.878465 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.878456 2584 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 03:48:33.878514 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.878506 2584 kubelet.go:493] "Attempting to sync node with API server" Nov 04 03:48:33.878542 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.878518 2584 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 03:48:33.878542 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.878531 2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 03:48:33.878542 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.878540 2584 kubelet.go:405] "Adding apiserver pod source" Nov 04 03:48:33.878665 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.878547 2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 03:48:33.878897 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.878886 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 03:48:33.880710 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.880694 2584 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 03:48:33.885362 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.885347 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 03:48:33.886084 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.886072 2584 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 03:48:33.886132 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.886102 2584 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 03:48:33.886266 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.886255 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 03:48:33.886329 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.886271 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 03:48:33.886329 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.886277 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 03:48:33.886329 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.886282 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 03:48:33.886329 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.886289 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 03:48:33.886329 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.886294 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 03:48:33.886736 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.886727 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 03:48:33.886776 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.886737 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 03:48:33.886776 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.886745 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 03:48:33.886776 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.886751 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 03:48:33.886776 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.886761 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 03:48:33.886776 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.886772 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 03:48:33.888145 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.888135 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 03:48:33.888145 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.888145 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 03:48:33.892490 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.892476 2584 server.go:1263] "Started kubelet" Nov 04 03:48:33.892850 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.892823 2584 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 03:48:33.892976 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.892931 2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-9.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 04 03:48:33.893059 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.893015 2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 03:48:33.893114 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.893078 2584 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 03:48:33.893138 ip-10-0-138-9 systemd[1]: Started Kubernetes Kubelet. Nov 04 03:48:33.894726 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.894708 2584 server.go:317] "Adding debug handlers to kubelet server" Nov 04 03:48:33.895510 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.895493 2584 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 03:48:33.898579 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:33.898548 2584 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 04 03:48:33.899347 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:33.899330 2584 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 04 03:48:33.899560 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:33.899543 2584 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-9.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 04 03:48:33.900133 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:33.899359 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-9.us-east-2.compute.internal.18e4baf533671162 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-9.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-9.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 03:48:33.892446562 +0000 UTC m=+0.348671350,LastTimestamp:2026-11-04 03:48:33.892446562 +0000 UTC m=+0.348671350,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-9.us-east-2.compute.internal,}" Nov 04 03:48:33.900582 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.900568 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 03:48:33.901992 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.901976 2584 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 03:48:33.902099 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:33.902074 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-9.us-east-2.compute.internal\" not found" Nov 04 03:48:33.902476 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.902457 2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 03:48:33.902476 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.902458 2584 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 03:48:33.902476 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.902484 2584 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 03:48:33.902682 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.902628 2584 reconstruct.go:97] "Volume reconstruction finished" Nov 04 03:48:33.902682 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.902635 2584 reconciler.go:29] "Reconciler: start to sync state" Nov 04 03:48:33.903076 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.903061 2584 factory.go:153] Registering CRI-O factory Nov 04 03:48:33.903143 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.903079 2584 factory.go:223] Registration of the crio container factory successfully Nov 04 03:48:33.903192 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.903158 2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 03:48:33.903192 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.903165 2584 factory.go:55] Registering systemd factory Nov 04 03:48:33.903192 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.903173 2584 factory.go:223] Registration of the systemd container factory successfully Nov 04 03:48:33.903192 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.903189 2584 factory.go:103] Registering Raw factory Nov 04 03:48:33.903366 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.903205 2584 manager.go:1199] Started watching for new ooms in manager Nov 04 03:48:33.905443 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:33.905415 2584 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-9.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 04 03:48:33.905528 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.905477 2584 manager.go:319] Starting recovery of all containers Nov 04 03:48:33.908316 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:33.908294 2584 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 04 03:48:33.917254 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.915858 2584 manager.go:324] Recovery completed Nov 04 03:48:33.921123 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.921110 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:48:33.922817 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.922794 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:48:33.922874 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.922831 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:48:33.922874 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.922844 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 03:48:33.923412 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.923399 2584 cpu_manager.go:226] "Starting" policy="none" Nov 04 03:48:33.923412 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.923410 2584 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 03:48:33.923501 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.923427 2584 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 03:48:33.923501 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.923412 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:48:33.924123 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.924110 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:48:33.924178 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.924134 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:48:33.924178 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.924146 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 03:48:33.925532 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:33.925472 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-9.us-east-2.compute.internal.18e4baf5353679f2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-9.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-9.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-9.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 03:48:33.922816498 +0000 UTC m=+0.379041287,LastTimestamp:2026-11-04 03:48:33.922816498 +0000 UTC m=+0.379041287,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-9.us-east-2.compute.internal,}" Nov 04 03:48:33.925649 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.925597 2584 policy_none.go:50] "Start" Nov 04 03:48:33.925693 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.925657 2584 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 03:48:33.925693 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.925673 2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 03:48:33.927696 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.927682 2584 policy_none.go:44] "Start" Nov 04 03:48:33.940780 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:33.940715 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-9.us-east-2.compute.internal.18e4baf53536cebf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-9.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-9.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-9.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 03:48:33.922838207 +0000 UTC m=+0.379062998,LastTimestamp:2026-11-04 03:48:33.922838207 +0000 UTC m=+0.379062998,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-9.us-east-2.compute.internal,}" Nov 04 03:48:33.956384 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.956367 2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-97v5f" Nov 04 03:48:33.959033 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:33.958971 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-9.us-east-2.compute.internal.18e4baf53536f77b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-9.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-9.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-9.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 03:48:33.922848635 +0000 UTC m=+0.379073424,LastTimestamp:2026-11-04 03:48:33.922848635 +0000 UTC m=+0.379073424,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-9.us-east-2.compute.internal,}" Nov 04 03:48:33.982835 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.969516 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-97v5f" Nov 04 03:48:33.982835 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.970230 2584 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 03:48:33.982835 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.973674 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 03:48:33.982835 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.973863 2584 manager.go:341] "Starting Device Plugin manager" Nov 04 03:48:33.982835 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:33.974326 2584 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 03:48:33.982835 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.974343 2584 server.go:87] "Starting device plugin registration server" Nov 04 03:48:33.982835 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.974623 2584 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 03:48:33.982835 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.974638 2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 03:48:33.982835 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.974819 2584 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 03:48:33.982835 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.974890 2584 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 03:48:33.982835 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.974898 2584 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 03:48:33.982835 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.975304 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 03:48:33.982835 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.975319 2584 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 03:48:33.982835 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.975340 2584 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 03:48:33.982835 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:33.975370 2584 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 03:48:33.982835 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:33.975379 2584 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 04 03:48:33.982835 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:33.975415 2584 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-9.us-east-2.compute.internal\" not found" Nov 04 03:48:33.982835 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:33.975559 2584 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-9.us-east-2.compute.internal.18e4baf5353679f2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-9.us-east-2.compute.internal.18e4baf5353679f2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-9.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-9.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-9.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 03:48:33.922816498 +0000 UTC m=+0.379041287,LastTimestamp:2026-11-04 03:48:33.924121776 +0000 UTC m=+0.380346564,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-9.us-east-2.compute.internal,}" Nov 04 03:48:33.989456 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:33.989441 2584 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:48:34.075651 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.075594 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:48:34.075651 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.075612 2584 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal"] Nov 04 03:48:34.075795 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.075680 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:48:34.077066 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.077046 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:48:34.077153 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.077081 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:48:34.077153 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.077096 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 03:48:34.077153 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.077054 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:48:34.077153 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.077152 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:48:34.077295 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.077162 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 03:48:34.077295 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.077182 2584 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:34.079406 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.079394 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:48:34.079537 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.079524 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:34.079570 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.079550 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:48:34.080177 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.080161 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:48:34.080177 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.080180 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:48:34.080300 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.080186 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:48:34.080300 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.080190 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 03:48:34.080300 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.080206 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:48:34.080300 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.080222 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 03:48:34.082209 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.082197 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:34.082267 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.082219 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:48:34.083022 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.083008 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:48:34.083084 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.083033 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:48:34.083084 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.083042 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 03:48:34.084628 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.084613 2584 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:34.084676 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:34.084635 2584 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-9.us-east-2.compute.internal\": node \"ip-10-0-138-9.us-east-2.compute.internal\" not found" Nov 04 03:48:34.104042 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.104023 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1d9b8a1217c8d6e24202560e65c65d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal\" (UID: \"c1d9b8a1217c8d6e24202560e65c65d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:34.106239 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:34.106224 2584 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-9.us-east-2.compute.internal\" not found" node="ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:34.108252 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:34.108237 2584 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-9.us-east-2.compute.internal\" not found" node="ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:34.116343 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:34.116327 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-9.us-east-2.compute.internal\" not found" Nov 04 03:48:34.204827 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.204790 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1d9b8a1217c8d6e24202560e65c65d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal\" (UID: \"c1d9b8a1217c8d6e24202560e65c65d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:34.204827 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.204829 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1d9b8a1217c8d6e24202560e65c65d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal\" (UID: \"c1d9b8a1217c8d6e24202560e65c65d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:34.205005 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.204860 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37008f3824659b029a90a4ae33f9a950-config\") pod \"kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal\" (UID: \"37008f3824659b029a90a4ae33f9a950\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:34.205005 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.204882 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1d9b8a1217c8d6e24202560e65c65d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal\" (UID: \"c1d9b8a1217c8d6e24202560e65c65d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:34.216919 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:34.216898 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-9.us-east-2.compute.internal\" not found" Nov 04 03:48:34.305321 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.305298 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1d9b8a1217c8d6e24202560e65c65d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal\" (UID: \"c1d9b8a1217c8d6e24202560e65c65d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:34.305384 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.305324 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37008f3824659b029a90a4ae33f9a950-config\") pod \"kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal\" (UID: \"37008f3824659b029a90a4ae33f9a950\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:34.305384 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.305350 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37008f3824659b029a90a4ae33f9a950-config\") pod \"kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal\" (UID: \"37008f3824659b029a90a4ae33f9a950\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:34.305447 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.305389 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1d9b8a1217c8d6e24202560e65c65d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal\" (UID: \"c1d9b8a1217c8d6e24202560e65c65d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:34.317394 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:34.317370 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-9.us-east-2.compute.internal\" not found" Nov 04 03:48:34.409145 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.409085 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:34.411267 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.411252 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:34.418233 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:34.418216 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-9.us-east-2.compute.internal\" not found" Nov 04 03:48:34.518783 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:34.518754 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-9.us-east-2.compute.internal\" not found" Nov 04 03:48:34.619260 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:34.619227 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-9.us-east-2.compute.internal\" not found" Nov 04 03:48:34.719969 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:34.719905 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-9.us-east-2.compute.internal\" not found" Nov 04 03:48:34.820387 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:34.820357 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-9.us-east-2.compute.internal\" not found" Nov 04 03:48:34.839520 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.839494 2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 04 03:48:34.839660 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:34.839636 2584 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 03:48:36.537702 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:36.537682 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-9.us-east-2.compute.internal\" not found" Nov 04 03:48:36.591678 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:36.591485 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 03:43:35 +0000 UTC" deadline="2028-08-20 14:44:23.386131252 +0000 UTC" Nov 04 03:48:36.591678 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:36.591650 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15730h55m46.794485547s" Nov 04 03:48:36.637894 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:36.637871 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-9.us-east-2.compute.internal\" not found" Nov 04 03:48:36.657333 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:36.657307 2584 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:48:36.699814 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:36.699788 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:48:36.700908 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:36.700887 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:48:36.700908 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:36.700929 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:48:36.701100 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:36.700941 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 03:48:36.738502 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:36.738477 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-9.us-east-2.compute.internal\" not found" Nov 04 03:48:36.771423 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:36.771390 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37008f3824659b029a90a4ae33f9a950.slice/crio-0991dbab3a77a90594ed04c9b3281fe729c4d76ca82aff5cd740d5937bc7219e WatchSource:0}: Error finding container 0991dbab3a77a90594ed04c9b3281fe729c4d76ca82aff5cd740d5937bc7219e: Status 404 returned error can't find the container with id 0991dbab3a77a90594ed04c9b3281fe729c4d76ca82aff5cd740d5937bc7219e Nov 04 03:48:36.771813 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:36.771793 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d9b8a1217c8d6e24202560e65c65d7.slice/crio-02ac35b4cb7404a4fb3d5ae493b982826d38cb4f4ee19f82583104436f253f62 WatchSource:0}: Error finding container 02ac35b4cb7404a4fb3d5ae493b982826d38cb4f4ee19f82583104436f253f62: Status 404 returned error can't find the container with id 02ac35b4cb7404a4fb3d5ae493b982826d38cb4f4ee19f82583104436f253f62 Nov 04 03:48:36.776773 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:36.776759 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 03:48:36.793178 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:36.793155 2584 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:48:36.819189 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:36.819164 2584 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:36.830632 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:36.830608 2584 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 03:48:36.831489 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:36.831478 2584 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:48:36.845327 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:36.845283 2584 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 03:48:36.858426 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:36.858410 2584 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:48:37.432836 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.432810 2584 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:48:37.495221 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.495197 2584 apiserver.go:52] "Watching apiserver" Nov 04 03:48:37.503695 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.503666 2584 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 03:48:37.504042 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.504020 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-c8xnp","openshift-multus/network-metrics-daemon-mwzdx","openshift-multus/multus-fsbpl","openshift-network-diagnostics/network-check-target-pbsd4","kube-system/global-pull-secret-syncer-59fsx","kube-system/konnectivity-agent-jvqw2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl","openshift-multus/multus-additional-cni-plugins-2pkqx","openshift-image-registry/node-ca-jkpjj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal","openshift-network-operator/iptables-alerter-28k9n","openshift-ovn-kubernetes/ovnkube-node-vxkvk","kube-system/kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-hl8q9"] Nov 04 03:48:37.506645 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.506600 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.508756 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.508737 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.511475 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.510914 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 03:48:37.511475 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.510947 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 03:48:37.511475 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.511008 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 03:48:37.511475 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.511020 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 03:48:37.511475 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.511049 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 03:48:37.511475 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.511112 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.512930 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.512898 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-twz4j\"" Nov 04 03:48:37.513123 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.513107 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 03:48:37.513194 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.513178 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-75zvg\"" Nov 04 03:48:37.513247 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.513230 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:48:37.513332 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.513303 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:37.513393 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:37.513361 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbsd4" podUID="391e5382-7119-4085-9939-cd81fc8b2d73" Nov 04 03:48:37.514309 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.514290 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 03:48:37.514480 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.514463 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-78gsl\"" Nov 04 03:48:37.515323 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.515304 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-28k9n" Nov 04 03:48:37.517360 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.517342 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 04 03:48:37.518215 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.518197 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jvqw2" Nov 04 03:48:37.520408 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.520387 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jkpjj" Nov 04 03:48:37.522826 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.522806 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.525221 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.525202 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c8xnp" Nov 04 03:48:37.527946 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.527909 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-systemd\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.528037 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.527957 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-lib-modules\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.528037 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.527981 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-host\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.528037 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528002 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgv6q\" (UniqueName: \"kubernetes.io/projected/2d275592-23f8-4edc-aff7-016930318629-kube-api-access-kgv6q\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.528037 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528026 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq2fj\" (UniqueName: \"kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj\") pod \"network-check-target-pbsd4\" (UID: \"391e5382-7119-4085-9939-cd81fc8b2d73\") " pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:37.528175 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528061 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/795d3139-8d19-4852-bd9c-3c60b01a97f6-serviceca\") pod \"node-ca-jkpjj\" (UID: \"795d3139-8d19-4852-bd9c-3c60b01a97f6\") " pod="openshift-image-registry/node-ca-jkpjj" Nov 04 03:48:37.528175 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528096 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d25872c4-1eee-48b8-a4be-9384dae267ec-iptables-alerter-script\") pod \"iptables-alerter-28k9n\" (UID: \"d25872c4-1eee-48b8-a4be-9384dae267ec\") " pod="openshift-network-operator/iptables-alerter-28k9n" Nov 04 03:48:37.528175 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528127 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-sys\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.528175 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528167 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/93149be5-0d1f-4b08-b33e-d2e43bac726c-konnectivity-ca\") pod \"konnectivity-agent-jvqw2\" (UID: \"93149be5-0d1f-4b08-b33e-d2e43bac726c\") " pod="kube-system/konnectivity-agent-jvqw2" Nov 04 03:48:37.528317 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528192 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-system-cni-dir\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.528317 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528211 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-cni-binary-copy\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.528317 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528234 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-run-netns\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.528317 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528266 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-modprobe-d\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.528317 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528301 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-kubernetes\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.528317 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528315 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-var-lib-kubelet\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.528595 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528329 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d275592-23f8-4edc-aff7-016930318629-tmp\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.528595 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528351 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/93149be5-0d1f-4b08-b33e-d2e43bac726c-agent-certs\") pod \"konnectivity-agent-jvqw2\" (UID: \"93149be5-0d1f-4b08-b33e-d2e43bac726c\") " pod="kube-system/konnectivity-agent-jvqw2" Nov 04 03:48:37.528595 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528385 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-var-lib-cni-multus\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.528595 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528407 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:37.528595 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528418 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-conf-dir\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.528595 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528449 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-daemon-config\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.528595 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:37.528478 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59fsx" podUID="26bb99fd-d223-4642-b147-b22bebd00835" Nov 04 03:48:37.528595 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528491 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-sysctl-d\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.528595 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528541 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-run\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.528595 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528563 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnklb\" (UniqueName: \"kubernetes.io/projected/795d3139-8d19-4852-bd9c-3c60b01a97f6-kube-api-access-wnklb\") pod \"node-ca-jkpjj\" (UID: \"795d3139-8d19-4852-bd9c-3c60b01a97f6\") " pod="openshift-image-registry/node-ca-jkpjj" Nov 04 03:48:37.529064 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528634 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-system-cni-dir\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.529064 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528689 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-cnibin\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.529064 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528720 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7a99c19-45c9-4294-b2af-42d20306ff28-cni-binary-copy\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.529064 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528744 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-run-k8s-cni-cncf-io\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.529064 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528770 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.529064 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528810 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-cni-dir\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.529064 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528856 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-socket-dir-parent\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.529064 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528882 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-var-lib-cni-bin\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.529064 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528938 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d25872c4-1eee-48b8-a4be-9384dae267ec-host-slash\") pod \"iptables-alerter-28k9n\" (UID: \"d25872c4-1eee-48b8-a4be-9384dae267ec\") " pod="openshift-network-operator/iptables-alerter-28k9n" Nov 04 03:48:37.529064 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.528963 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfk6m\" (UniqueName: \"kubernetes.io/projected/d25872c4-1eee-48b8-a4be-9384dae267ec-kube-api-access-dfk6m\") pod \"iptables-alerter-28k9n\" (UID: \"d25872c4-1eee-48b8-a4be-9384dae267ec\") " pod="openshift-network-operator/iptables-alerter-28k9n" Nov 04 03:48:37.529064 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.529018 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-sysctl-conf\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.529501 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.529059 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2d275592-23f8-4edc-aff7-016930318629-etc-tuned\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.529501 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.529105 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/795d3139-8d19-4852-bd9c-3c60b01a97f6-host\") pod \"node-ca-jkpjj\" (UID: \"795d3139-8d19-4852-bd9c-3c60b01a97f6\") " pod="openshift-image-registry/node-ca-jkpjj" Nov 04 03:48:37.529501 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.529141 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.529501 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.529167 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7b4p\" (UniqueName: \"kubernetes.io/projected/f7a99c19-45c9-4294-b2af-42d20306ff28-kube-api-access-w7b4p\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.529501 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.529191 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dcsj\" (UniqueName: \"kubernetes.io/projected/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-kube-api-access-6dcsj\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.529501 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.529214 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-var-lib-kubelet\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.529501 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.529272 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-sysconfig\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.529501 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.529312 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-cnibin\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.529501 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.529337 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.529501 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.529361 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-hostroot\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.529501 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.529382 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-run-multus-certs\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.529501 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.529405 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-etc-kubernetes\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.531084 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.531063 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.533426 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.533233 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:37.533426 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:37.533348 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzdx" podUID="9ab83f7b-3396-4cc7-875e-c2cc62696ff5" Nov 04 03:48:37.534276 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.534254 2584 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 03:48:37.553264 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.553241 2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gqzcz" Nov 04 03:48:37.574525 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.574499 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gqzcz" Nov 04 03:48:37.596948 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.596881 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" event={"ID":"c1d9b8a1217c8d6e24202560e65c65d7","Type":"ContainerStarted","Data":"02ac35b4cb7404a4fb3d5ae493b982826d38cb4f4ee19f82583104436f253f62"} Nov 04 03:48:37.598451 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.598425 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal" event={"ID":"37008f3824659b029a90a4ae33f9a950","Type":"ContainerStarted","Data":"0991dbab3a77a90594ed04c9b3281fe729c4d76ca82aff5cd740d5937bc7219e"} Nov 04 03:48:37.611257 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.611236 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 03:48:37.611358 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.611308 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 03:48:37.611358 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.611315 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 03:48:37.611358 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.611341 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 03:48:37.611501 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.611390 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 03:48:37.611501 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.611243 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 03:48:37.611501 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.611344 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 03:48:37.611501 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.611391 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 03:48:37.611501 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.611430 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4ttlw\"" Nov 04 03:48:37.611793 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.611766 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-qdtd9\"" Nov 04 03:48:37.612026 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.612001 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 03:48:37.612119 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.612048 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 03:48:37.612119 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.612084 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 03:48:37.612217 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.612135 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 03:48:37.612217 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.612192 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 03:48:37.612311 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.612301 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tl6ng\"" Nov 04 03:48:37.612359 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.612336 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:48:37.612419 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.612405 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 03:48:37.612497 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.612484 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j5cz2\"" Nov 04 03:48:37.612550 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.612521 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 03:48:37.612662 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.612647 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 03:48:37.612753 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.612736 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x8rgd\"" Nov 04 03:48:37.612857 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.612756 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 03:48:37.612857 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.612775 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 03:48:37.612857 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.612839 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m5nwx\"" Nov 04 03:48:37.620438 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.620418 2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 03:48:37.630560 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.630539 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-conf-dir\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.630645 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.630571 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.630645 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.630595 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19970c28-9dee-4377-8e98-1fc65473d3e1-ovn-node-metrics-cert\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.630645 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.630621 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-var-lib-cni-bin\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.630800 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.630644 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-conf-dir\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.630800 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.630721 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnklb\" (UniqueName: \"kubernetes.io/projected/795d3139-8d19-4852-bd9c-3c60b01a97f6-kube-api-access-wnklb\") pod \"node-ca-jkpjj\" (UID: \"795d3139-8d19-4852-bd9c-3c60b01a97f6\") " pod="openshift-image-registry/node-ca-jkpjj" Nov 04 03:48:37.630800 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.630734 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-var-lib-cni-bin\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.630800 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.630751 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-system-cni-dir\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.630800 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.630774 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-cnibin\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.630800 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.630797 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7a99c19-45c9-4294-b2af-42d20306ff28-cni-binary-copy\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.631004 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.630822 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12382e2-90c4-45bf-bdd8-27653073cc5d-tmp-dir\") pod \"node-resolver-c8xnp\" (UID: \"c12382e2-90c4-45bf-bdd8-27653073cc5d\") " pod="openshift-dns/node-resolver-c8xnp" Nov 04 03:48:37.631004 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.630849 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.631004 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.630872 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-socket-dir-parent\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.631004 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.630898 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfk6m\" (UniqueName: \"kubernetes.io/projected/d25872c4-1eee-48b8-a4be-9384dae267ec-kube-api-access-dfk6m\") pod \"iptables-alerter-28k9n\" (UID: \"d25872c4-1eee-48b8-a4be-9384dae267ec\") " pod="openshift-network-operator/iptables-alerter-28k9n" Nov 04 03:48:37.631004 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.630939 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-run-ovn\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.631004 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.630994 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19970c28-9dee-4377-8e98-1fc65473d3e1-env-overrides\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.631284 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.631027 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8q89\" (UniqueName: \"kubernetes.io/projected/19970c28-9dee-4377-8e98-1fc65473d3e1-kube-api-access-n8q89\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.631284 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.631157 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-system-cni-dir\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.631376 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.631293 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.631431 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.631407 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-socket-dir-parent\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.631581 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.631560 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7a99c19-45c9-4294-b2af-42d20306ff28-cni-binary-copy\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.631639 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.631624 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26bb99fd-d223-4642-b147-b22bebd00835-dbus\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:37.631673 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.631662 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-cnibin\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.631766 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.631751 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2d275592-23f8-4edc-aff7-016930318629-etc-tuned\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.631846 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.631830 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-device-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.631913 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.631869 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dcsj\" (UniqueName: \"kubernetes.io/projected/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-kube-api-access-6dcsj\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.631987 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.631930 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-kubelet\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.631987 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.631966 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-node-log\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.632059 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.631998 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-cnibin\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.632059 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.632022 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-hostroot\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.632119 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.632075 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-cni-netd\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.632119 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.632099 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-systemd\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.632183 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.632123 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-lib-modules\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.632183 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.632148 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2fj\" (UniqueName: \"kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj\") pod \"network-check-target-pbsd4\" (UID: \"391e5382-7119-4085-9939-cd81fc8b2d73\") " pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:37.632183 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.632171 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/795d3139-8d19-4852-bd9c-3c60b01a97f6-serviceca\") pod \"node-ca-jkpjj\" (UID: \"795d3139-8d19-4852-bd9c-3c60b01a97f6\") " pod="openshift-image-registry/node-ca-jkpjj" Nov 04 03:48:37.632305 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.632195 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26bb99fd-d223-4642-b147-b22bebd00835-kubelet-config\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:37.632305 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.632220 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-sys\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.632305 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.632245 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/93149be5-0d1f-4b08-b33e-d2e43bac726c-konnectivity-ca\") pod \"konnectivity-agent-jvqw2\" (UID: \"93149be5-0d1f-4b08-b33e-d2e43bac726c\") " pod="kube-system/konnectivity-agent-jvqw2" Nov 04 03:48:37.632397 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.632358 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-cni-binary-copy\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.632397 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.632384 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-run-netns\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.632453 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.632394 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-cnibin\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.632453 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.632419 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-registration-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.632513 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.632481 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-systemd\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.632605 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.632591 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-lib-modules\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.632934 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.632894 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/93149be5-0d1f-4b08-b33e-d2e43bac726c-konnectivity-ca\") pod \"konnectivity-agent-jvqw2\" (UID: \"93149be5-0d1f-4b08-b33e-d2e43bac726c\") " pod="kube-system/konnectivity-agent-jvqw2" Nov 04 03:48:37.632990 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.632963 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-log-socket\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.633029 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.632999 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7b4p\" (UniqueName: \"kubernetes.io/projected/f7a99c19-45c9-4294-b2af-42d20306ff28-kube-api-access-w7b4p\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.633059 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633029 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb5w2\" (UniqueName: \"kubernetes.io/projected/7714ee23-de1d-4e5d-88ba-cce91e08ba23-kube-api-access-fb5w2\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.633098 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633059 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d275592-23f8-4edc-aff7-016930318629-tmp\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.633098 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633088 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/93149be5-0d1f-4b08-b33e-d2e43bac726c-agent-certs\") pod \"konnectivity-agent-jvqw2\" (UID: \"93149be5-0d1f-4b08-b33e-d2e43bac726c\") " pod="kube-system/konnectivity-agent-jvqw2" Nov 04 03:48:37.633201 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633185 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-daemon-config\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.633238 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633203 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/795d3139-8d19-4852-bd9c-3c60b01a97f6-serviceca\") pod \"node-ca-jkpjj\" (UID: \"795d3139-8d19-4852-bd9c-3c60b01a97f6\") " pod="openshift-image-registry/node-ca-jkpjj" Nov 04 03:48:37.633238 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633212 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-slash\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.633238 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633228 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19970c28-9dee-4377-8e98-1fc65473d3e1-ovnkube-script-lib\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.633321 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633244 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d25872c4-1eee-48b8-a4be-9384dae267ec-host-slash\") pod \"iptables-alerter-28k9n\" (UID: \"d25872c4-1eee-48b8-a4be-9384dae267ec\") " pod="openshift-network-operator/iptables-alerter-28k9n" Nov 04 03:48:37.633321 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633262 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-sysctl-d\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.633321 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633283 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-run\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.633321 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633299 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-run-k8s-cni-cncf-io\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.633321 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633314 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-systemd-units\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.633477 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633329 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-run-netns\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.633477 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633331 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.633477 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633383 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d25872c4-1eee-48b8-a4be-9384dae267ec-host-slash\") pod \"iptables-alerter-28k9n\" (UID: \"d25872c4-1eee-48b8-a4be-9384dae267ec\") " pod="openshift-network-operator/iptables-alerter-28k9n" Nov 04 03:48:37.633570 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633473 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-sysctl-d\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.633570 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633522 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-run\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.633570 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633261 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-sys\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.633706 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633627 2584 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 03:48:37.634271 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.633984 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-hostroot\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.634271 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634044 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-run-k8s-cni-cncf-io\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.634271 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634044 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-cni-binary-copy\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.634271 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634212 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs\") pod \"network-metrics-daemon-mwzdx\" (UID: \"9ab83f7b-3396-4cc7-875e-c2cc62696ff5\") " pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:37.634271 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634247 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-cni-dir\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.634556 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634311 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-sysctl-conf\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.634556 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634352 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/795d3139-8d19-4852-bd9c-3c60b01a97f6-host\") pod \"node-ca-jkpjj\" (UID: \"795d3139-8d19-4852-bd9c-3c60b01a97f6\") " pod="openshift-image-registry/node-ca-jkpjj" Nov 04 03:48:37.634556 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634377 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.634556 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634409 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-cni-dir\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.634556 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634430 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-socket-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.634556 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634458 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-var-lib-kubelet\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.634556 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634484 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-etc-openvswitch\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.634556 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634533 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-sysctl-conf\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.634556 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634547 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c12382e2-90c4-45bf-bdd8-27653073cc5d-hosts-file\") pod \"node-resolver-c8xnp\" (UID: \"c12382e2-90c4-45bf-bdd8-27653073cc5d\") " pod="openshift-dns/node-resolver-c8xnp" Nov 04 03:48:37.634997 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634576 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-sysconfig\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.634997 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634589 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-daemon-config\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.634997 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634603 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.634997 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634646 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-var-lib-kubelet\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.634997 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634632 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-run-multus-certs\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.634997 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634721 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-run-multus-certs\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.634997 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634747 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/795d3139-8d19-4852-bd9c-3c60b01a97f6-host\") pod \"node-ca-jkpjj\" (UID: \"795d3139-8d19-4852-bd9c-3c60b01a97f6\") " pod="openshift-image-registry/node-ca-jkpjj" Nov 04 03:48:37.634997 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634756 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-etc-kubernetes\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.634997 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634783 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-var-lib-openvswitch\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.634997 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634795 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-sysconfig\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.634997 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634836 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-run-openvswitch\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.634997 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634861 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:37.634997 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634900 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vf5x\" (UniqueName: \"kubernetes.io/projected/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-kube-api-access-4vf5x\") pod \"network-metrics-daemon-mwzdx\" (UID: \"9ab83f7b-3396-4cc7-875e-c2cc62696ff5\") " pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:37.634997 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634942 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-host\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.634997 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634969 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgv6q\" (UniqueName: \"kubernetes.io/projected/2d275592-23f8-4edc-aff7-016930318629-kube-api-access-kgv6q\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.634997 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.634994 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d25872c4-1eee-48b8-a4be-9384dae267ec-iptables-alerter-script\") pod \"iptables-alerter-28k9n\" (UID: \"d25872c4-1eee-48b8-a4be-9384dae267ec\") " pod="openshift-network-operator/iptables-alerter-28k9n" Nov 04 03:48:37.635742 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635006 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.635742 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635064 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj5tf\" (UniqueName: \"kubernetes.io/projected/c12382e2-90c4-45bf-bdd8-27653073cc5d-kube-api-access-lj5tf\") pod \"node-resolver-c8xnp\" (UID: \"c12382e2-90c4-45bf-bdd8-27653073cc5d\") " pod="openshift-dns/node-resolver-c8xnp" Nov 04 03:48:37.635742 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635104 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-system-cni-dir\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.635742 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635131 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-etc-selinux\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.635742 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635154 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-sys-fs\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.635742 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635211 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-run-netns\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.635742 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635236 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-run-systemd\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.635742 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635251 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.635742 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635263 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-cni-bin\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.635742 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635289 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19970c28-9dee-4377-8e98-1fc65473d3e1-ovnkube-config\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.635742 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635302 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-etc-kubernetes\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.635742 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635314 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.635742 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635345 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-modprobe-d\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.635742 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635371 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-kubernetes\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.635742 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635397 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-var-lib-kubelet\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.635742 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635425 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-var-lib-cni-multus\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.636321 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635537 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d25872c4-1eee-48b8-a4be-9384dae267ec-iptables-alerter-script\") pod \"iptables-alerter-28k9n\" (UID: \"d25872c4-1eee-48b8-a4be-9384dae267ec\") " pod="openshift-network-operator/iptables-alerter-28k9n" Nov 04 03:48:37.636321 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635552 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-host\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.636321 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635574 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-var-lib-cni-multus\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.636321 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635600 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-system-cni-dir\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.636321 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635621 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-var-lib-kubelet\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.636321 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635645 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-kubernetes\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.636321 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.635644 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-modprobe-d\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.639671 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.639650 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d275592-23f8-4edc-aff7-016930318629-tmp\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.640020 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.639971 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/93149be5-0d1f-4b08-b33e-d2e43bac726c-agent-certs\") pod \"konnectivity-agent-jvqw2\" (UID: \"93149be5-0d1f-4b08-b33e-d2e43bac726c\") " pod="kube-system/konnectivity-agent-jvqw2" Nov 04 03:48:37.641454 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.641429 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2d275592-23f8-4edc-aff7-016930318629-etc-tuned\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.643591 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:37.643004 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 03:48:37.643591 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:37.643028 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 03:48:37.643591 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:37.643043 2584 projected.go:196] Error preparing data for projected volume kube-api-access-mq2fj for pod openshift-network-diagnostics/network-check-target-pbsd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:48:37.643591 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:37.643116 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj podName:391e5382-7119-4085-9939-cd81fc8b2d73 nodeName:}" failed. No retries permitted until 2026-11-04 03:48:38.143084377 +0000 UTC m=+2.982994885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mq2fj" (UniqueName: "kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj") pod "network-check-target-pbsd4" (UID: "391e5382-7119-4085-9939-cd81fc8b2d73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:48:37.645253 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.645231 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfk6m\" (UniqueName: \"kubernetes.io/projected/d25872c4-1eee-48b8-a4be-9384dae267ec-kube-api-access-dfk6m\") pod \"iptables-alerter-28k9n\" (UID: \"d25872c4-1eee-48b8-a4be-9384dae267ec\") " pod="openshift-network-operator/iptables-alerter-28k9n" Nov 04 03:48:37.646943 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.646444 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnklb\" (UniqueName: \"kubernetes.io/projected/795d3139-8d19-4852-bd9c-3c60b01a97f6-kube-api-access-wnklb\") pod \"node-ca-jkpjj\" (UID: \"795d3139-8d19-4852-bd9c-3c60b01a97f6\") " pod="openshift-image-registry/node-ca-jkpjj" Nov 04 03:48:37.649904 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.649655 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7b4p\" (UniqueName: \"kubernetes.io/projected/f7a99c19-45c9-4294-b2af-42d20306ff28-kube-api-access-w7b4p\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.650015 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.649967 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgv6q\" (UniqueName: \"kubernetes.io/projected/2d275592-23f8-4edc-aff7-016930318629-kube-api-access-kgv6q\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.655988 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.654412 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dcsj\" (UniqueName: \"kubernetes.io/projected/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-kube-api-access-6dcsj\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.736539 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736461 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-run-netns\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.736539 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736506 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-run-systemd\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.736539 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736529 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-cni-bin\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.736887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736554 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19970c28-9dee-4377-8e98-1fc65473d3e1-ovnkube-config\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.736887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736579 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.736887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736589 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-cni-bin\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.736887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736607 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.736887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736591 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-run-systemd\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.736887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736631 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19970c28-9dee-4377-8e98-1fc65473d3e1-ovn-node-metrics-cert\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.736887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736657 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12382e2-90c4-45bf-bdd8-27653073cc5d-tmp-dir\") pod \"node-resolver-c8xnp\" (UID: \"c12382e2-90c4-45bf-bdd8-27653073cc5d\") " pod="openshift-dns/node-resolver-c8xnp" Nov 04 03:48:37.736887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736660 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.736887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736685 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-run-ovn\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.736887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736707 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19970c28-9dee-4377-8e98-1fc65473d3e1-env-overrides\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.736887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736730 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8q89\" (UniqueName: \"kubernetes.io/projected/19970c28-9dee-4377-8e98-1fc65473d3e1-kube-api-access-n8q89\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.736887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736733 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.736887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736752 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26bb99fd-d223-4642-b147-b22bebd00835-dbus\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:37.736887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736778 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-device-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.736887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736802 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-kubelet\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.736887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736828 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-node-log\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.736887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736850 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-cni-netd\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.737712 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736886 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26bb99fd-d223-4642-b147-b22bebd00835-kubelet-config\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:37.737712 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736912 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-registration-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.737712 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736952 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-log-socket\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.737712 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736976 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb5w2\" (UniqueName: \"kubernetes.io/projected/7714ee23-de1d-4e5d-88ba-cce91e08ba23-kube-api-access-fb5w2\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.737712 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737001 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-slash\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.737712 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737023 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19970c28-9dee-4377-8e98-1fc65473d3e1-ovnkube-script-lib\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.737712 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737035 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12382e2-90c4-45bf-bdd8-27653073cc5d-tmp-dir\") pod \"node-resolver-c8xnp\" (UID: \"c12382e2-90c4-45bf-bdd8-27653073cc5d\") " pod="openshift-dns/node-resolver-c8xnp" Nov 04 03:48:37.737712 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737052 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-systemd-units\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.737712 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737093 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.737712 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737123 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs\") pod \"network-metrics-daemon-mwzdx\" (UID: \"9ab83f7b-3396-4cc7-875e-c2cc62696ff5\") " pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:37.737712 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737130 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-run-ovn\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.737712 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737154 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-socket-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.737712 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737180 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-etc-openvswitch\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.737712 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737198 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.737712 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737199 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19970c28-9dee-4377-8e98-1fc65473d3e1-ovnkube-config\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.737712 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737204 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c12382e2-90c4-45bf-bdd8-27653073cc5d-hosts-file\") pod \"node-resolver-c8xnp\" (UID: \"c12382e2-90c4-45bf-bdd8-27653073cc5d\") " pod="openshift-dns/node-resolver-c8xnp" Nov 04 03:48:37.737712 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737241 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-etc-openvswitch\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.738340 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737250 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-var-lib-openvswitch\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.738340 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.736550 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-run-netns\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.738340 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737267 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-run-openvswitch\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.738340 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737283 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:37.738340 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:37.737283 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:48:37.738340 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737298 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vf5x\" (UniqueName: \"kubernetes.io/projected/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-kube-api-access-4vf5x\") pod \"network-metrics-daemon-mwzdx\" (UID: \"9ab83f7b-3396-4cc7-875e-c2cc62696ff5\") " pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:37.738340 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737316 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj5tf\" (UniqueName: \"kubernetes.io/projected/c12382e2-90c4-45bf-bdd8-27653073cc5d-kube-api-access-lj5tf\") pod \"node-resolver-c8xnp\" (UID: \"c12382e2-90c4-45bf-bdd8-27653073cc5d\") " pod="openshift-dns/node-resolver-c8xnp" Nov 04 03:48:37.738340 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:37.737345 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs podName:9ab83f7b-3396-4cc7-875e-c2cc62696ff5 nodeName:}" failed. No retries permitted until 2026-11-04 03:48:38.23732774 +0000 UTC m=+3.077238249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs") pod "network-metrics-daemon-mwzdx" (UID: "9ab83f7b-3396-4cc7-875e-c2cc62696ff5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:48:37.738340 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737365 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-etc-selinux\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.738340 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737392 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-sys-fs\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.738340 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737470 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-var-lib-openvswitch\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.738340 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737473 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-sys-fs\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.738340 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737478 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19970c28-9dee-4377-8e98-1fc65473d3e1-env-overrides\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.738340 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737097 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-systemd-units\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.738340 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737497 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-run-openvswitch\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.738340 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737531 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c12382e2-90c4-45bf-bdd8-27653073cc5d-hosts-file\") pod \"node-resolver-c8xnp\" (UID: \"c12382e2-90c4-45bf-bdd8-27653073cc5d\") " pod="openshift-dns/node-resolver-c8xnp" Nov 04 03:48:37.738340 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:37.737540 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 03:48:37.739183 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:37.737582 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret podName:26bb99fd-d223-4642-b147-b22bebd00835 nodeName:}" failed. No retries permitted until 2026-11-04 03:48:38.23757034 +0000 UTC m=+3.077480825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret") pod "global-pull-secret-syncer-59fsx" (UID: "26bb99fd-d223-4642-b147-b22bebd00835") : object "kube-system"/"original-pull-secret" not registered Nov 04 03:48:37.739183 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737617 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-socket-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.739183 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737752 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-etc-selinux\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.739183 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737759 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26bb99fd-d223-4642-b147-b22bebd00835-dbus\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:37.739183 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737809 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26bb99fd-d223-4642-b147-b22bebd00835-kubelet-config\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:37.739183 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737813 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-slash\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.739183 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737860 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-device-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.739183 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737905 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-kubelet\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.739183 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737963 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-node-log\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.739183 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.737990 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-log-socket\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.739183 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.738008 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-cni-netd\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.739183 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.738044 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-registration-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.739183 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.738331 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19970c28-9dee-4377-8e98-1fc65473d3e1-ovnkube-script-lib\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.740393 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.740368 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19970c28-9dee-4377-8e98-1fc65473d3e1-ovn-node-metrics-cert\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.750212 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.750181 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj5tf\" (UniqueName: \"kubernetes.io/projected/c12382e2-90c4-45bf-bdd8-27653073cc5d-kube-api-access-lj5tf\") pod \"node-resolver-c8xnp\" (UID: \"c12382e2-90c4-45bf-bdd8-27653073cc5d\") " pod="openshift-dns/node-resolver-c8xnp" Nov 04 03:48:37.750328 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.750211 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8q89\" (UniqueName: \"kubernetes.io/projected/19970c28-9dee-4377-8e98-1fc65473d3e1-kube-api-access-n8q89\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.750715 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.750689 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vf5x\" (UniqueName: \"kubernetes.io/projected/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-kube-api-access-4vf5x\") pod \"network-metrics-daemon-mwzdx\" (UID: \"9ab83f7b-3396-4cc7-875e-c2cc62696ff5\") " pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:37.752908 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.752863 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb5w2\" (UniqueName: \"kubernetes.io/projected/7714ee23-de1d-4e5d-88ba-cce91e08ba23-kube-api-access-fb5w2\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.837283 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.837245 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:48:37.841283 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.841257 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:48:37.845240 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.845217 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fsbpl" Nov 04 03:48:37.851325 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.851165 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-28k9n" Nov 04 03:48:37.851784 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:37.851761 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ced90a_6b3d_4553_b05f_d4ec50e6af9b.slice/crio-93905c9baf281e2ae8e958a45a501766968cca88dcc1f69544fb23f30e845554 WatchSource:0}: Error finding container 93905c9baf281e2ae8e958a45a501766968cca88dcc1f69544fb23f30e845554: Status 404 returned error can't find the container with id 93905c9baf281e2ae8e958a45a501766968cca88dcc1f69544fb23f30e845554 Nov 04 03:48:37.853292 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:37.853270 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d275592_23f8_4edc_aff7_016930318629.slice/crio-28a344d57c8845001afb7553c6efa4c6b0de214265e5f2880bfa3510bb886043 WatchSource:0}: Error finding container 28a344d57c8845001afb7553c6efa4c6b0de214265e5f2880bfa3510bb886043: Status 404 returned error can't find the container with id 28a344d57c8845001afb7553c6efa4c6b0de214265e5f2880bfa3510bb886043 Nov 04 03:48:37.857072 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:37.856940 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a99c19_45c9_4294_b2af_42d20306ff28.slice/crio-342e443435b836d0cf9df45f36d28198960585e34a612e38b5283e83b250beab WatchSource:0}: Error finding container 342e443435b836d0cf9df45f36d28198960585e34a612e38b5283e83b250beab: Status 404 returned error can't find the container with id 342e443435b836d0cf9df45f36d28198960585e34a612e38b5283e83b250beab Nov 04 03:48:37.860324 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.860304 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jvqw2" Nov 04 03:48:37.863185 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:37.863165 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25872c4_1eee_48b8_a4be_9384dae267ec.slice/crio-435aab2da18b8024da0c2fdd7672a6661ac9dac1fd21dd6093a7c50323ae4c16 WatchSource:0}: Error finding container 435aab2da18b8024da0c2fdd7672a6661ac9dac1fd21dd6093a7c50323ae4c16: Status 404 returned error can't find the container with id 435aab2da18b8024da0c2fdd7672a6661ac9dac1fd21dd6093a7c50323ae4c16 Nov 04 03:48:37.867626 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.867608 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jkpjj" Nov 04 03:48:37.870133 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:37.870106 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93149be5_0d1f_4b08_b33e_d2e43bac726c.slice/crio-a26a5b00ae7f18543adc8f26e387393e0fbb02a58d5c7f5ed524ffad17eb4a4d WatchSource:0}: Error finding container a26a5b00ae7f18543adc8f26e387393e0fbb02a58d5c7f5ed524ffad17eb4a4d: Status 404 returned error can't find the container with id a26a5b00ae7f18543adc8f26e387393e0fbb02a58d5c7f5ed524ffad17eb4a4d Nov 04 03:48:37.877144 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:37.877126 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795d3139_8d19_4852_bd9c_3c60b01a97f6.slice/crio-918686dc6f7e96901c7733bc446a40b812cd1f44b39515659d8bdc3e61b526b8 WatchSource:0}: Error finding container 918686dc6f7e96901c7733bc446a40b812cd1f44b39515659d8bdc3e61b526b8: Status 404 returned error can't find the container with id 918686dc6f7e96901c7733bc446a40b812cd1f44b39515659d8bdc3e61b526b8 Nov 04 03:48:37.880662 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.880641 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:48:37.884198 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.884172 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c8xnp" Nov 04 03:48:37.891761 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:37.891740 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7714ee23_de1d_4e5d_88ba_cce91e08ba23.slice/crio-908140abe84c9bde75743f0962814b626593b42e67045aee092d5be0fe496d71 WatchSource:0}: Error finding container 908140abe84c9bde75743f0962814b626593b42e67045aee092d5be0fe496d71: Status 404 returned error can't find the container with id 908140abe84c9bde75743f0962814b626593b42e67045aee092d5be0fe496d71 Nov 04 03:48:37.894616 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:37.894599 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12382e2_90c4_45bf_bdd8_27653073cc5d.slice/crio-0a5f14aae34c9231e88c3a57c9892d6a604279f39f7f89a165f67896f845fc8a WatchSource:0}: Error finding container 0a5f14aae34c9231e88c3a57c9892d6a604279f39f7f89a165f67896f845fc8a: Status 404 returned error can't find the container with id 0a5f14aae34c9231e88c3a57c9892d6a604279f39f7f89a165f67896f845fc8a Nov 04 03:48:37.905087 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:37.905069 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:37.915239 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:48:37.915212 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19970c28_9dee_4377_8e98_1fc65473d3e1.slice/crio-fd59a627fdda9658fab68cef4d1850c1ae343b40131951cc6eebdbea3cc3f0ab WatchSource:0}: Error finding container fd59a627fdda9658fab68cef4d1850c1ae343b40131951cc6eebdbea3cc3f0ab: Status 404 returned error can't find the container with id fd59a627fdda9658fab68cef4d1850c1ae343b40131951cc6eebdbea3cc3f0ab Nov 04 03:48:38.241087 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:38.241055 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2fj\" (UniqueName: \"kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj\") pod \"network-check-target-pbsd4\" (UID: \"391e5382-7119-4085-9939-cd81fc8b2d73\") " pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:38.241241 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:38.241116 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs\") pod \"network-metrics-daemon-mwzdx\" (UID: \"9ab83f7b-3396-4cc7-875e-c2cc62696ff5\") " pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:38.241241 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:38.241151 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:38.241368 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:38.241241 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 03:48:38.241368 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:38.241265 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 03:48:38.241368 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:38.241278 2584 projected.go:196] Error preparing data for projected volume kube-api-access-mq2fj for pod openshift-network-diagnostics/network-check-target-pbsd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:48:38.241368 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:38.241283 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 03:48:38.241368 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:38.241340 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj podName:391e5382-7119-4085-9939-cd81fc8b2d73 nodeName:}" failed. No retries permitted until 2026-11-04 03:48:39.241320576 +0000 UTC m=+4.081231078 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mq2fj" (UniqueName: "kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj") pod "network-check-target-pbsd4" (UID: "391e5382-7119-4085-9939-cd81fc8b2d73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:48:38.241368 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:38.241361 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:48:38.241625 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:38.241392 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret podName:26bb99fd-d223-4642-b147-b22bebd00835 nodeName:}" failed. No retries permitted until 2026-11-04 03:48:39.241356023 +0000 UTC m=+4.081266513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret") pod "global-pull-secret-syncer-59fsx" (UID: "26bb99fd-d223-4642-b147-b22bebd00835") : object "kube-system"/"original-pull-secret" not registered Nov 04 03:48:38.241625 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:38.241411 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs podName:9ab83f7b-3396-4cc7-875e-c2cc62696ff5 nodeName:}" failed. No retries permitted until 2026-11-04 03:48:39.241401061 +0000 UTC m=+4.081311553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs") pod "network-metrics-daemon-mwzdx" (UID: "9ab83f7b-3396-4cc7-875e-c2cc62696ff5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:48:38.577813 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:38.577698 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 03:43:37 +0000 UTC" deadline="2028-04-08 16:20:46.493906692 +0000 UTC" Nov 04 03:48:38.577813 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:38.577769 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12516h32m7.916142599s" Nov 04 03:48:38.593727 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:38.593233 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:38.593727 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:38.593365 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbsd4" podUID="391e5382-7119-4085-9939-cd81fc8b2d73" Nov 04 03:48:38.608192 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:38.608138 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" event={"ID":"7714ee23-de1d-4e5d-88ba-cce91e08ba23","Type":"ContainerStarted","Data":"908140abe84c9bde75743f0962814b626593b42e67045aee092d5be0fe496d71"} Nov 04 03:48:38.611189 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:38.611111 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jkpjj" event={"ID":"795d3139-8d19-4852-bd9c-3c60b01a97f6","Type":"ContainerStarted","Data":"918686dc6f7e96901c7733bc446a40b812cd1f44b39515659d8bdc3e61b526b8"} Nov 04 03:48:38.616430 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:38.616351 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jvqw2" event={"ID":"93149be5-0d1f-4b08-b33e-d2e43bac726c","Type":"ContainerStarted","Data":"a26a5b00ae7f18543adc8f26e387393e0fbb02a58d5c7f5ed524ffad17eb4a4d"} Nov 04 03:48:38.625823 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:38.625739 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-28k9n" event={"ID":"d25872c4-1eee-48b8-a4be-9384dae267ec","Type":"ContainerStarted","Data":"435aab2da18b8024da0c2fdd7672a6661ac9dac1fd21dd6093a7c50323ae4c16"} Nov 04 03:48:38.634848 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:38.634776 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" event={"ID":"2d275592-23f8-4edc-aff7-016930318629","Type":"ContainerStarted","Data":"28a344d57c8845001afb7553c6efa4c6b0de214265e5f2880bfa3510bb886043"} Nov 04 03:48:38.640524 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:38.640442 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pkqx" event={"ID":"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b","Type":"ContainerStarted","Data":"93905c9baf281e2ae8e958a45a501766968cca88dcc1f69544fb23f30e845554"} Nov 04 03:48:38.643420 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:38.643282 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" event={"ID":"19970c28-9dee-4377-8e98-1fc65473d3e1","Type":"ContainerStarted","Data":"fd59a627fdda9658fab68cef4d1850c1ae343b40131951cc6eebdbea3cc3f0ab"} Nov 04 03:48:38.646082 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:38.645972 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c8xnp" event={"ID":"c12382e2-90c4-45bf-bdd8-27653073cc5d","Type":"ContainerStarted","Data":"0a5f14aae34c9231e88c3a57c9892d6a604279f39f7f89a165f67896f845fc8a"} Nov 04 03:48:38.653070 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:38.653008 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fsbpl" event={"ID":"f7a99c19-45c9-4294-b2af-42d20306ff28","Type":"ContainerStarted","Data":"342e443435b836d0cf9df45f36d28198960585e34a612e38b5283e83b250beab"} Nov 04 03:48:39.252307 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:39.252276 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2fj\" (UniqueName: \"kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj\") pod \"network-check-target-pbsd4\" (UID: \"391e5382-7119-4085-9939-cd81fc8b2d73\") " pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:39.252494 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:39.252333 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs\") pod \"network-metrics-daemon-mwzdx\" (UID: \"9ab83f7b-3396-4cc7-875e-c2cc62696ff5\") " pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:39.252494 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:39.252368 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:39.252602 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:39.252504 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 03:48:39.252602 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:39.252560 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret podName:26bb99fd-d223-4642-b147-b22bebd00835 nodeName:}" failed. No retries permitted until 2026-11-04 03:48:41.252546536 +0000 UTC m=+6.092457039 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret") pod "global-pull-secret-syncer-59fsx" (UID: "26bb99fd-d223-4642-b147-b22bebd00835") : object "kube-system"/"original-pull-secret" not registered Nov 04 03:48:39.252724 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:39.252620 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 03:48:39.252724 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:39.252630 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 03:48:39.252724 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:39.252639 2584 projected.go:196] Error preparing data for projected volume kube-api-access-mq2fj for pod openshift-network-diagnostics/network-check-target-pbsd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:48:39.252724 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:39.252662 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj podName:391e5382-7119-4085-9939-cd81fc8b2d73 nodeName:}" failed. No retries permitted until 2026-11-04 03:48:41.252654947 +0000 UTC m=+6.092565433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mq2fj" (UniqueName: "kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj") pod "network-check-target-pbsd4" (UID: "391e5382-7119-4085-9939-cd81fc8b2d73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:48:39.252724 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:39.252698 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:48:39.252724 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:39.252725 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs podName:9ab83f7b-3396-4cc7-875e-c2cc62696ff5 nodeName:}" failed. No retries permitted until 2026-11-04 03:48:41.252717697 +0000 UTC m=+6.092628181 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs") pod "network-metrics-daemon-mwzdx" (UID: "9ab83f7b-3396-4cc7-875e-c2cc62696ff5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:48:39.578376 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:39.578326 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 03:43:37 +0000 UTC" deadline="2028-05-26 07:13:25.958840863 +0000 UTC" Nov 04 03:48:39.578376 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:39.578368 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13659h24m46.380477371s" Nov 04 03:48:39.594526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:39.594501 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:39.594671 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:39.594601 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59fsx" podUID="26bb99fd-d223-4642-b147-b22bebd00835" Nov 04 03:48:39.594987 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:39.594893 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:39.595089 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:39.595000 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzdx" podUID="9ab83f7b-3396-4cc7-875e-c2cc62696ff5" Nov 04 03:48:39.665294 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:39.665256 2584 generic.go:356] "Generic (PLEG): container finished" podID="c1d9b8a1217c8d6e24202560e65c65d7" containerID="0aac18d105a9e8bde98e426e49bb32667d1d4fe81a7505a750d6258dbc1e8ec3" exitCode=0 Nov 04 03:48:39.665448 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:39.665365 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" event={"ID":"c1d9b8a1217c8d6e24202560e65c65d7","Type":"ContainerDied","Data":"0aac18d105a9e8bde98e426e49bb32667d1d4fe81a7505a750d6258dbc1e8ec3"} Nov 04 03:48:39.688635 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:39.687814 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal" event={"ID":"37008f3824659b029a90a4ae33f9a950","Type":"ContainerStarted","Data":"c6c62fb6e965d78c0599cdb30727f127440fec7674bc7b16638221cbc5498f2f"} Nov 04 03:48:40.592553 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:40.592236 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:40.593002 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:40.592662 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbsd4" podUID="391e5382-7119-4085-9939-cd81fc8b2d73" Nov 04 03:48:40.717033 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:40.716977 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" event={"ID":"c1d9b8a1217c8d6e24202560e65c65d7","Type":"ContainerStarted","Data":"74c2662d4fa71d149b9f11e1b1828c2fcb7f031b738ce068de310cb729f3c63a"} Nov 04 03:48:40.739444 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:40.738597 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal" podStartSLOduration=4.73858028 podStartE2EDuration="4.73858028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:48:39.706253396 +0000 UTC m=+4.546163903" watchObservedRunningTime="2026-11-04 03:48:40.73858028 +0000 UTC m=+5.578490788" Nov 04 03:48:41.271856 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:41.271821 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs\") pod \"network-metrics-daemon-mwzdx\" (UID: \"9ab83f7b-3396-4cc7-875e-c2cc62696ff5\") " pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:41.272046 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:41.271879 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:41.272046 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:41.271952 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2fj\" (UniqueName: \"kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj\") pod \"network-check-target-pbsd4\" (UID: \"391e5382-7119-4085-9939-cd81fc8b2d73\") " pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:41.272146 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:41.272090 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 03:48:41.272146 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:41.272107 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 03:48:41.272146 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:41.272119 2584 projected.go:196] Error preparing data for projected volume kube-api-access-mq2fj for pod openshift-network-diagnostics/network-check-target-pbsd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:48:41.272298 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:41.272175 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj podName:391e5382-7119-4085-9939-cd81fc8b2d73 nodeName:}" failed. No retries permitted until 2026-11-04 03:48:45.272157656 +0000 UTC m=+10.112068157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mq2fj" (UniqueName: "kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj") pod "network-check-target-pbsd4" (UID: "391e5382-7119-4085-9939-cd81fc8b2d73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:48:41.272693 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:41.272671 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:48:41.272759 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:41.272749 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs podName:9ab83f7b-3396-4cc7-875e-c2cc62696ff5 nodeName:}" failed. No retries permitted until 2026-11-04 03:48:45.272730641 +0000 UTC m=+10.112641131 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs") pod "network-metrics-daemon-mwzdx" (UID: "9ab83f7b-3396-4cc7-875e-c2cc62696ff5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:48:41.272822 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:41.272813 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 03:48:41.272865 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:41.272853 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret podName:26bb99fd-d223-4642-b147-b22bebd00835 nodeName:}" failed. No retries permitted until 2026-11-04 03:48:45.272839077 +0000 UTC m=+10.112749577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret") pod "global-pull-secret-syncer-59fsx" (UID: "26bb99fd-d223-4642-b147-b22bebd00835") : object "kube-system"/"original-pull-secret" not registered Nov 04 03:48:41.594498 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:41.594465 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:41.595000 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:41.594599 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59fsx" podUID="26bb99fd-d223-4642-b147-b22bebd00835" Nov 04 03:48:41.595000 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:41.594980 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:41.595120 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:41.595090 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzdx" podUID="9ab83f7b-3396-4cc7-875e-c2cc62696ff5" Nov 04 03:48:41.715514 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:41.715469 2584 generic.go:356] "Generic (PLEG): container finished" podID="a1ced90a-6b3d-4553-b05f-d4ec50e6af9b" containerID="6b3b988aa3691a5eeeb37794b1c0958a74ba2d97398c7b483b2433cffca61ab5" exitCode=0 Nov 04 03:48:41.716630 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:41.716602 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pkqx" event={"ID":"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b","Type":"ContainerDied","Data":"6b3b988aa3691a5eeeb37794b1c0958a74ba2d97398c7b483b2433cffca61ab5"} Nov 04 03:48:41.743885 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:41.743842 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" podStartSLOduration=5.7438289860000005 podStartE2EDuration="5.743828986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:48:40.739408938 +0000 UTC m=+5.579319446" watchObservedRunningTime="2026-11-04 03:48:41.743828986 +0000 UTC m=+6.583739526" Nov 04 03:48:42.592378 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:42.592339 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:42.592568 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:42.592470 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbsd4" podUID="391e5382-7119-4085-9939-cd81fc8b2d73" Nov 04 03:48:42.723944 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:42.723431 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jvqw2" event={"ID":"93149be5-0d1f-4b08-b33e-d2e43bac726c","Type":"ContainerStarted","Data":"2fe898aea5c8c669966bf714aa0445f13e33d53191d1483e5dd0664061c6e314"} Nov 04 03:48:43.603119 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:43.603089 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:43.603285 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:43.603135 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:43.603345 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:43.603283 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzdx" podUID="9ab83f7b-3396-4cc7-875e-c2cc62696ff5" Nov 04 03:48:43.603760 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:43.603738 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59fsx" podUID="26bb99fd-d223-4642-b147-b22bebd00835" Nov 04 03:48:43.729086 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:43.729047 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jkpjj" event={"ID":"795d3139-8d19-4852-bd9c-3c60b01a97f6","Type":"ContainerStarted","Data":"3cd79e06d5cc292db4e917326da07e2e27958f2bfe577faefe08f970ca59be1d"} Nov 04 03:48:43.745695 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:43.745626 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jvqw2" podStartSLOduration=4.832599923 podStartE2EDuration="8.745613213s" totalImagesPullingTime="3.91301329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:48:42.741906804 +0000 UTC m=+7.581817314" watchObservedRunningTime="2026-11-04 03:48:43.745613213 +0000 UTC m=+8.585523719" Nov 04 03:48:44.592148 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:44.592116 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:44.592334 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:44.592249 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbsd4" podUID="391e5382-7119-4085-9939-cd81fc8b2d73" Nov 04 03:48:44.739090 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:44.738998 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" event={"ID":"7714ee23-de1d-4e5d-88ba-cce91e08ba23","Type":"ContainerStarted","Data":"6e5e626d62e6388ec1dce6f4c72a88d132fa9b5b1a715a7c674bc375c7259ca1"} Nov 04 03:48:45.306184 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:45.306138 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:45.306323 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:45.306236 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2fj\" (UniqueName: \"kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj\") pod \"network-check-target-pbsd4\" (UID: \"391e5382-7119-4085-9939-cd81fc8b2d73\") " pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:45.306323 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:45.306283 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs\") pod \"network-metrics-daemon-mwzdx\" (UID: \"9ab83f7b-3396-4cc7-875e-c2cc62696ff5\") " pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:45.306663 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:45.306430 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:48:45.306663 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:45.306502 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs podName:9ab83f7b-3396-4cc7-875e-c2cc62696ff5 nodeName:}" failed. No retries permitted until 2026-11-04 03:48:53.306482569 +0000 UTC m=+18.146393061 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs") pod "network-metrics-daemon-mwzdx" (UID: "9ab83f7b-3396-4cc7-875e-c2cc62696ff5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:48:45.306957 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:45.306938 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 03:48:45.307027 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:45.306998 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret podName:26bb99fd-d223-4642-b147-b22bebd00835 nodeName:}" failed. No retries permitted until 2026-11-04 03:48:53.306980178 +0000 UTC m=+18.146890668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret") pod "global-pull-secret-syncer-59fsx" (UID: "26bb99fd-d223-4642-b147-b22bebd00835") : object "kube-system"/"original-pull-secret" not registered Nov 04 03:48:45.307101 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:45.307088 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 03:48:45.307146 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:45.307107 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 03:48:45.307146 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:45.307120 2584 projected.go:196] Error preparing data for projected volume kube-api-access-mq2fj for pod openshift-network-diagnostics/network-check-target-pbsd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:48:45.307218 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:45.307158 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj podName:391e5382-7119-4085-9939-cd81fc8b2d73 nodeName:}" failed. No retries permitted until 2026-11-04 03:48:53.307146231 +0000 UTC m=+18.147056750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mq2fj" (UniqueName: "kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj") pod "network-check-target-pbsd4" (UID: "391e5382-7119-4085-9939-cd81fc8b2d73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:48:45.485860 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:45.485804 2584 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 03:48:45.593386 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:45.592778 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:45.593386 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:45.592885 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59fsx" podUID="26bb99fd-d223-4642-b147-b22bebd00835" Nov 04 03:48:45.593386 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:45.593227 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:45.593386 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:45.593333 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzdx" podUID="9ab83f7b-3396-4cc7-875e-c2cc62696ff5" Nov 04 03:48:45.604397 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:45.604058 2584 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T03:48:45.485856374Z","UUID":"286d6427-bbab-4db3-9406-a8a02aa86433","Handler":null,"Name":"","Endpoint":""} Nov 04 03:48:45.605757 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:45.605731 2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 03:48:45.605757 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:45.605757 2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 03:48:45.744644 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:45.743909 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fsbpl" event={"ID":"f7a99c19-45c9-4294-b2af-42d20306ff28","Type":"ContainerStarted","Data":"1a02c291a7f5bab387b3bbdb70502918f3881a4b4a6a6db8dd51ae77e6bf78c5"} Nov 04 03:48:45.747438 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:45.747410 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" event={"ID":"7714ee23-de1d-4e5d-88ba-cce91e08ba23","Type":"ContainerStarted","Data":"106516e085959e756c7e66565fbb382c469f57ec6c749b4888965c9bdd844942"} Nov 04 03:48:45.764833 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:45.764785 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jkpjj" podStartSLOduration=5.571005348 podStartE2EDuration="10.764765247s" totalImagesPullingTime="5.193759899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:48:43.746425213 +0000 UTC m=+8.586335724" watchObservedRunningTime="2026-11-04 03:48:45.764765247 +0000 UTC m=+10.604675754" Nov 04 03:48:45.765373 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:45.765334 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-fsbpl" podStartSLOduration=3.552080898 podStartE2EDuration="10.765325066s" totalImagesPullingTime="7.213244168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:48:45.764759888 +0000 UTC m=+10.604670397" watchObservedRunningTime="2026-11-04 03:48:45.765325066 +0000 UTC m=+10.605235572" Nov 04 03:48:46.268535 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:46.268493 2584 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jvqw2" Nov 04 03:48:46.270063 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:46.270039 2584 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jvqw2" Nov 04 03:48:46.592792 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:46.592763 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:46.592983 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:46.592865 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbsd4" podUID="391e5382-7119-4085-9939-cd81fc8b2d73" Nov 04 03:48:46.752807 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:46.752772 2584 generic.go:356] "Generic (PLEG): container finished" podID="a1ced90a-6b3d-4553-b05f-d4ec50e6af9b" containerID="f61e71aafb6a895d7bf75d0a378ab632014e5e925ea34dfed6a0b188a5f2ffd6" exitCode=0 Nov 04 03:48:46.753365 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:46.752875 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pkqx" event={"ID":"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b","Type":"ContainerDied","Data":"f61e71aafb6a895d7bf75d0a378ab632014e5e925ea34dfed6a0b188a5f2ffd6"} Nov 04 03:48:46.758004 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:46.757983 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c8xnp" event={"ID":"c12382e2-90c4-45bf-bdd8-27653073cc5d","Type":"ContainerStarted","Data":"d981195821347002f875398f6136d95c71b5cfe6aac7c106150f1ddac5dbee15"} Nov 04 03:48:47.592330 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:47.592006 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:47.592330 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:47.592111 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:47.592462 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:47.592346 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59fsx" podUID="26bb99fd-d223-4642-b147-b22bebd00835" Nov 04 03:48:47.592462 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:47.592432 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzdx" podUID="9ab83f7b-3396-4cc7-875e-c2cc62696ff5" Nov 04 03:48:47.762891 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:47.762816 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" event={"ID":"7714ee23-de1d-4e5d-88ba-cce91e08ba23","Type":"ContainerStarted","Data":"6f4d03817d0fb216fd410959c3d39b65f7073ef738d4c48eafcb9c410c86af85"} Nov 04 03:48:47.764563 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:47.764527 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-28k9n" event={"ID":"d25872c4-1eee-48b8-a4be-9384dae267ec","Type":"ContainerStarted","Data":"a0ff68c67f800bda120fb8ba3b89cbbc3d523c3300f767cb7d8f4981459e3b94"} Nov 04 03:48:47.767037 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:47.767007 2584 generic.go:356] "Generic (PLEG): container finished" podID="a1ced90a-6b3d-4553-b05f-d4ec50e6af9b" containerID="5c94be59ea87683b74e08b1e0652c8cd15bb80c8e0da0857e0111e5f4ac699dd" exitCode=0 Nov 04 03:48:47.767444 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:47.767421 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pkqx" event={"ID":"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b","Type":"ContainerDied","Data":"5c94be59ea87683b74e08b1e0652c8cd15bb80c8e0da0857e0111e5f4ac699dd"} Nov 04 03:48:47.784760 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:47.784720 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" podStartSLOduration=3.479501181 podStartE2EDuration="11.784708196s" totalImagesPullingTime="8.305207015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:48:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:48:47.784465742 +0000 UTC m=+12.624376249" watchObservedRunningTime="2026-11-04 03:48:47.784708196 +0000 UTC m=+12.624618703" Nov 04 03:48:47.785250 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:47.785214 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-c8xnp" podStartSLOduration=3.85942205 podStartE2EDuration="11.785206337s" totalImagesPullingTime="7.925784287s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:48:46.793668169 +0000 UTC m=+11.633578678" watchObservedRunningTime="2026-11-04 03:48:47.785206337 +0000 UTC m=+12.625116844" Nov 04 03:48:47.800612 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:47.800575 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-28k9n" podStartSLOduration=4.622934651 podStartE2EDuration="12.800562745s" totalImagesPullingTime="8.177628094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:48:47.800401685 +0000 UTC m=+12.640312192" watchObservedRunningTime="2026-11-04 03:48:47.800562745 +0000 UTC m=+12.640473251" Nov 04 03:48:48.592893 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:48.592858 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:48.593076 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:48.593002 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbsd4" podUID="391e5382-7119-4085-9939-cd81fc8b2d73" Nov 04 03:48:48.770306 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:48.770268 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" event={"ID":"2d275592-23f8-4edc-aff7-016930318629","Type":"ContainerStarted","Data":"320b98a17c84f5f98f33a39570355bb315f76dfee96e7d2d3ab5c8453b081a15"} Nov 04 03:48:48.790794 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:48.790745 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" podStartSLOduration=3.828493914 podStartE2EDuration="13.7907303s" totalImagesPullingTime="9.962236386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:48:48.79044171 +0000 UTC m=+13.630352213" watchObservedRunningTime="2026-11-04 03:48:48.7907303 +0000 UTC m=+13.630640804" Nov 04 03:48:49.259822 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:49.259734 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jvqw2" Nov 04 03:48:49.259984 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:49.259883 2584 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 03:48:49.260413 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:49.260384 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jvqw2" Nov 04 03:48:49.592116 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:49.592079 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:49.592302 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:49.592217 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzdx" podUID="9ab83f7b-3396-4cc7-875e-c2cc62696ff5" Nov 04 03:48:49.592302 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:49.592249 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:49.592411 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:49.592325 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59fsx" podUID="26bb99fd-d223-4642-b147-b22bebd00835" Nov 04 03:48:49.775711 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:49.775675 2584 generic.go:356] "Generic (PLEG): container finished" podID="a1ced90a-6b3d-4553-b05f-d4ec50e6af9b" containerID="3d7f6f1b17c3aeb902e2a9fc40375fd5d4149a3332c6d70ff29b9c9e5f2aa994" exitCode=0 Nov 04 03:48:49.776391 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:49.775774 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pkqx" event={"ID":"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b","Type":"ContainerDied","Data":"3d7f6f1b17c3aeb902e2a9fc40375fd5d4149a3332c6d70ff29b9c9e5f2aa994"} Nov 04 03:48:50.592308 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:50.592278 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:50.592473 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:50.592396 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbsd4" podUID="391e5382-7119-4085-9939-cd81fc8b2d73" Nov 04 03:48:51.592895 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:51.592856 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:51.593577 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:51.593011 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59fsx" podUID="26bb99fd-d223-4642-b147-b22bebd00835" Nov 04 03:48:51.593577 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:51.593460 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:51.593577 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:51.593543 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzdx" podUID="9ab83f7b-3396-4cc7-875e-c2cc62696ff5" Nov 04 03:48:52.592847 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:52.592677 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:52.593054 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:52.592910 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbsd4" podUID="391e5382-7119-4085-9939-cd81fc8b2d73" Nov 04 03:48:52.784835 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:52.784806 2584 generic.go:356] "Generic (PLEG): container finished" podID="a1ced90a-6b3d-4553-b05f-d4ec50e6af9b" containerID="386748536b10856fc1342a279fd31808cbe188b6b7d4cb276c8ba739dba32fc5" exitCode=0 Nov 04 03:48:52.784952 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:52.784883 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pkqx" event={"ID":"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b","Type":"ContainerDied","Data":"386748536b10856fc1342a279fd31808cbe188b6b7d4cb276c8ba739dba32fc5"} Nov 04 03:48:52.786617 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:52.786593 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" event={"ID":"19970c28-9dee-4377-8e98-1fc65473d3e1","Type":"ContainerStarted","Data":"a2f922e2c693505c804de199622abf4c9310e476f262d7bf4ec70ced464edd3f"} Nov 04 03:48:53.369339 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:53.369312 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2fj\" (UniqueName: \"kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj\") pod \"network-check-target-pbsd4\" (UID: \"391e5382-7119-4085-9939-cd81fc8b2d73\") " pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:53.369499 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:53.369350 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs\") pod \"network-metrics-daemon-mwzdx\" (UID: \"9ab83f7b-3396-4cc7-875e-c2cc62696ff5\") " pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:53.369499 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:53.369374 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:53.369499 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:53.369473 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 03:48:53.369499 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:53.369487 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 03:48:53.369700 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:53.369507 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 03:48:53.369700 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:53.369520 2584 projected.go:196] Error preparing data for projected volume kube-api-access-mq2fj for pod openshift-network-diagnostics/network-check-target-pbsd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:48:53.369700 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:53.369527 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret podName:26bb99fd-d223-4642-b147-b22bebd00835 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:09.369509682 +0000 UTC m=+34.209420167 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret") pod "global-pull-secret-syncer-59fsx" (UID: "26bb99fd-d223-4642-b147-b22bebd00835") : object "kube-system"/"original-pull-secret" not registered Nov 04 03:48:53.369700 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:53.369553 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj podName:391e5382-7119-4085-9939-cd81fc8b2d73 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:09.369543816 +0000 UTC m=+34.209454315 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mq2fj" (UniqueName: "kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj") pod "network-check-target-pbsd4" (UID: "391e5382-7119-4085-9939-cd81fc8b2d73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:48:53.369700 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:53.369486 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:48:53.369700 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:53.369588 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs podName:9ab83f7b-3396-4cc7-875e-c2cc62696ff5 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:09.369579908 +0000 UTC m=+34.209490408 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs") pod "network-metrics-daemon-mwzdx" (UID: "9ab83f7b-3396-4cc7-875e-c2cc62696ff5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:48:53.592585 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:53.592557 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:53.592585 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:53.592582 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:53.592769 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:53.592672 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzdx" podUID="9ab83f7b-3396-4cc7-875e-c2cc62696ff5" Nov 04 03:48:53.592808 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:53.592786 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59fsx" podUID="26bb99fd-d223-4642-b147-b22bebd00835" Nov 04 03:48:53.791551 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:53.791479 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" event={"ID":"19970c28-9dee-4377-8e98-1fc65473d3e1","Type":"ContainerStarted","Data":"95153c4fd5e90c63eb18adbb02eb47cd695e091d50d9e8f6d6337bf7d8df7331"} Nov 04 03:48:53.791551 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:53.791544 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" event={"ID":"19970c28-9dee-4377-8e98-1fc65473d3e1","Type":"ContainerStarted","Data":"142b6bc81b6951786e3adc90ba2f2916b1f2928f5773030c40531428b2511608"} Nov 04 03:48:53.791959 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:53.791577 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" event={"ID":"19970c28-9dee-4377-8e98-1fc65473d3e1","Type":"ContainerStarted","Data":"28476dd3480939cc181a965d1fb73350aed7e2393d3a7cdf90625a588c358f3e"} Nov 04 03:48:53.791959 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:53.791617 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" event={"ID":"19970c28-9dee-4377-8e98-1fc65473d3e1","Type":"ContainerStarted","Data":"faa4d7fd8bb98f8d02844380bd07af496ae12eea975f5d55ae2698c54f9f274c"} Nov 04 03:48:53.791959 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:53.791649 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" event={"ID":"19970c28-9dee-4377-8e98-1fc65473d3e1","Type":"ContainerStarted","Data":"d0f29f62ac8f49cb1711a2881a4acbbbe8f949027757c7e48f55a51f60a381c2"} Nov 04 03:48:53.794234 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:53.794210 2584 generic.go:356] "Generic (PLEG): container finished" podID="a1ced90a-6b3d-4553-b05f-d4ec50e6af9b" containerID="93f5523a3a2e128d54a1f428fa567b19341efd7fd1b097e2ef6ed7b749966f2a" exitCode=0 Nov 04 03:48:53.794327 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:53.794242 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pkqx" event={"ID":"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b","Type":"ContainerDied","Data":"93f5523a3a2e128d54a1f428fa567b19341efd7fd1b097e2ef6ed7b749966f2a"} Nov 04 03:48:54.592519 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:54.592478 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:54.592665 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:54.592604 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbsd4" podUID="391e5382-7119-4085-9939-cd81fc8b2d73" Nov 04 03:48:54.799612 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:54.799580 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pkqx" event={"ID":"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b","Type":"ContainerStarted","Data":"7110fcb4ee9060f4fc0b587e6fdd26030fd7e986ce2b7eee775b532e9bd26595"} Nov 04 03:48:54.826566 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:54.826526 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2pkqx" podStartSLOduration=8.977800977 podStartE2EDuration="19.826513992s" totalImagesPullingTime="10.848713015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:48:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:48:54.826194288 +0000 UTC m=+19.666104795" watchObservedRunningTime="2026-11-04 03:48:54.826513992 +0000 UTC m=+19.666424495" Nov 04 03:48:55.592564 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:55.592540 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:55.592725 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:55.592616 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzdx" podUID="9ab83f7b-3396-4cc7-875e-c2cc62696ff5" Nov 04 03:48:55.592725 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:55.592709 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:55.592862 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:55.592838 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59fsx" podUID="26bb99fd-d223-4642-b147-b22bebd00835" Nov 04 03:48:55.804018 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:55.803986 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" event={"ID":"19970c28-9dee-4377-8e98-1fc65473d3e1","Type":"ContainerStarted","Data":"9c7700476a50dc0af2202ea85c37e9c71535143e8afe97464c9461e0fe4b4a21"} Nov 04 03:48:56.592709 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:56.592679 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:56.592855 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:56.592764 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbsd4" podUID="391e5382-7119-4085-9939-cd81fc8b2d73" Nov 04 03:48:57.592337 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:57.592302 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:57.593016 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:57.592444 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59fsx" podUID="26bb99fd-d223-4642-b147-b22bebd00835" Nov 04 03:48:57.593016 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:57.592490 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:57.593016 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:57.592584 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzdx" podUID="9ab83f7b-3396-4cc7-875e-c2cc62696ff5" Nov 04 03:48:57.811712 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:57.811674 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" event={"ID":"19970c28-9dee-4377-8e98-1fc65473d3e1","Type":"ContainerStarted","Data":"dab3e9dd027c412ae47983c7366a394e0eb4b6345bb3747a73fdf435f5d74649"} Nov 04 03:48:57.811959 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:57.811910 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:57.812071 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:57.811991 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:57.812071 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:57.812029 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:57.826887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:57.826864 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:57.827030 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:57.827003 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:48:57.847395 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:57.847319 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" podStartSLOduration=7.083007611 podStartE2EDuration="21.84730416s" totalImagesPullingTime="14.764296549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:48:57.845530175 +0000 UTC m=+22.685440681" watchObservedRunningTime="2026-11-04 03:48:57.84730416 +0000 UTC m=+22.687214698" Nov 04 03:48:58.592793 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:58.592522 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:58.593163 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:58.592942 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbsd4" podUID="391e5382-7119-4085-9939-cd81fc8b2d73" Nov 04 03:48:58.984555 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:58.984472 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-59fsx"] Nov 04 03:48:58.984706 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:58.984580 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:48:58.984706 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:58.984658 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59fsx" podUID="26bb99fd-d223-4642-b147-b22bebd00835" Nov 04 03:48:58.991694 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:58.991670 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pbsd4"] Nov 04 03:48:58.991825 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:58.991766 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:48:58.991883 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:58.991847 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbsd4" podUID="391e5382-7119-4085-9939-cd81fc8b2d73" Nov 04 03:48:58.992348 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:58.992319 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mwzdx"] Nov 04 03:48:58.992452 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:48:58.992437 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:48:58.992572 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:48:58.992551 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzdx" podUID="9ab83f7b-3396-4cc7-875e-c2cc62696ff5" Nov 04 03:49:00.592838 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:00.592810 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:49:00.593312 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:00.592904 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:49:00.593312 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:00.593010 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbsd4" podUID="391e5382-7119-4085-9939-cd81fc8b2d73" Nov 04 03:49:00.593312 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:00.592908 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzdx" podUID="9ab83f7b-3396-4cc7-875e-c2cc62696ff5" Nov 04 03:49:00.593312 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:00.592909 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:49:00.593312 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:00.593088 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59fsx" podUID="26bb99fd-d223-4642-b147-b22bebd00835" Nov 04 03:49:02.591818 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:02.591786 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:49:02.591818 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:02.591816 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:49:02.592241 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:02.591912 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzdx" podUID="9ab83f7b-3396-4cc7-875e-c2cc62696ff5" Nov 04 03:49:02.592241 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:02.591962 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbsd4" podUID="391e5382-7119-4085-9939-cd81fc8b2d73" Nov 04 03:49:02.592241 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:02.591993 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:49:02.592241 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:02.592079 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59fsx" podUID="26bb99fd-d223-4642-b147-b22bebd00835" Nov 04 03:49:04.592234 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:04.592195 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:49:04.592234 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:04.592222 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:49:04.592868 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:04.592320 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:49:04.592868 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:04.592338 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbsd4" podUID="391e5382-7119-4085-9939-cd81fc8b2d73" Nov 04 03:49:04.592868 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:04.592388 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59fsx" podUID="26bb99fd-d223-4642-b147-b22bebd00835" Nov 04 03:49:04.592868 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:04.592481 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzdx" podUID="9ab83f7b-3396-4cc7-875e-c2cc62696ff5" Nov 04 03:49:05.020719 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.020695 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeReady" Nov 04 03:49:05.020887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.020813 2584 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 04 03:49:05.058168 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.058134 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qq74s"] Nov 04 03:49:05.066274 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.066249 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8v2sd"] Nov 04 03:49:05.066385 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.066327 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qq74s" Nov 04 03:49:05.070309 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.070292 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2hhp2"] Nov 04 03:49:05.070420 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.070407 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8v2sd" Nov 04 03:49:05.074732 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.074338 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wc5zp"] Nov 04 03:49:05.074732 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.074425 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2hhp2" Nov 04 03:49:05.077770 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.077752 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-78ccf8f879-s4sx4"] Nov 04 03:49:05.077886 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.077875 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wc5zp" Nov 04 03:49:05.080509 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.080495 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r"] Nov 04 03:49:05.080618 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.080605 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:05.084035 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.083991 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-gm5s7"] Nov 04 03:49:05.084207 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.084189 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:05.091017 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.090993 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8"] Nov 04 03:49:05.095361 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.093853 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 04 03:49:05.095361 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.094744 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-q9vm5\"" Nov 04 03:49:05.095361 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.094851 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.096890 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.096870 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 04 03:49:05.097033 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.096940 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lvdkb\"" Nov 04 03:49:05.099775 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.099379 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 04 03:49:05.100989 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.100965 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz"] Nov 04 03:49:05.101114 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.101061 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" Nov 04 03:49:05.108456 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.108320 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rr5bv"] Nov 04 03:49:05.108551 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.108482 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:05.115146 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.114251 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm"] Nov 04 03:49:05.115146 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.114616 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:05.116933 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.116893 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vjhfk"] Nov 04 03:49:05.117028 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.117011 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" Nov 04 03:49:05.121220 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.121198 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6448ff747c-dlpnw"] Nov 04 03:49:05.121454 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.121431 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vjhfk" Nov 04 03:49:05.124592 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.124575 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8648b894c6-t5rf2"] Nov 04 03:49:05.124780 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.124761 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.128543 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.128525 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-5xh74"] Nov 04 03:49:05.128708 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.128690 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.132025 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.131978 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8v2sd"] Nov 04 03:49:05.132294 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.132278 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wc5zp"] Nov 04 03:49:05.132389 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.132312 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r"] Nov 04 03:49:05.132389 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.132334 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-78ccf8f879-s4sx4"] Nov 04 03:49:05.132389 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.132355 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qq74s"] Nov 04 03:49:05.132389 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.132375 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rr5bv"] Nov 04 03:49:05.132615 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.132397 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm"] Nov 04 03:49:05.132615 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.132420 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-gm5s7"] Nov 04 03:49:05.132615 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.132440 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8"] Nov 04 03:49:05.132615 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.132462 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pvrd4"] Nov 04 03:49:05.132799 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.132745 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5xh74" Nov 04 03:49:05.136388 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.136366 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-chk4t"] Nov 04 03:49:05.136484 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.136471 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:05.140308 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.140290 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-5xh74"] Nov 04 03:49:05.140398 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.140369 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-chk4t" Nov 04 03:49:05.140398 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.140382 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2hhp2"] Nov 04 03:49:05.140471 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.140417 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vjhfk"] Nov 04 03:49:05.140471 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.140442 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pvrd4"] Nov 04 03:49:05.140545 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.140471 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz"] Nov 04 03:49:05.140545 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.140497 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8648b894c6-t5rf2"] Nov 04 03:49:05.140545 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.140518 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6448ff747c-dlpnw"] Nov 04 03:49:05.140545 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.140541 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-chk4t"] Nov 04 03:49:05.153886 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.153860 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b456f4f-071a-4e5f-b8b6-51c3ee790656-tmp\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.153997 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.153889 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhvtl\" (UniqueName: \"kubernetes.io/projected/1c2ac1f5-6e08-48e3-ac83-a55f9181f4be-kube-api-access-zhvtl\") pod \"kas-connection-checker-685895c7c9-wc5zp\" (UID: \"1c2ac1f5-6e08-48e3-ac83-a55f9181f4be\") " pod="kube-system/kas-connection-checker-685895c7c9-wc5zp" Nov 04 03:49:05.153997 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.153950 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knwrb\" (UniqueName: \"kubernetes.io/projected/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-kube-api-access-knwrb\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:05.154112 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.154088 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf4rk\" (UniqueName: \"kubernetes.io/projected/1f0a14d4-9771-4aa2-a6bd-87f5f73ee931-kube-api-access-sf4rk\") pod \"kas-connection-checker-685895c7c9-qq74s\" (UID: \"1f0a14d4-9771-4aa2-a6bd-87f5f73ee931\") " pod="kube-system/kas-connection-checker-685895c7c9-qq74s" Nov 04 03:49:05.154177 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.154119 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr88g\" (UniqueName: \"kubernetes.io/projected/27d72728-5623-41e0-bee5-a09937f9f87a-kube-api-access-kr88g\") pod \"kas-connection-checker-685895c7c9-8v2sd\" (UID: \"27d72728-5623-41e0-bee5-a09937f9f87a\") " pod="kube-system/kas-connection-checker-685895c7c9-8v2sd" Nov 04 03:49:05.154243 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.154224 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f37b227d-fc9c-4a81-85b1-62c04243ad83-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2hhp2\" (UID: \"f37b227d-fc9c-4a81-85b1-62c04243ad83\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2hhp2" Nov 04 03:49:05.154294 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.154258 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9fvw\" (UniqueName: \"kubernetes.io/projected/0df87f5d-31f4-4571-82b6-fe75c434ff27-kube-api-access-w9fvw\") pod \"cluster-monitoring-operator-884576c57-qmh6r\" (UID: \"0df87f5d-31f4-4571-82b6-fe75c434ff27\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:05.154294 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.154285 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:05.154380 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.154313 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-metrics-certs\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:05.154476 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.154448 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0df87f5d-31f4-4571-82b6-fe75c434ff27-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qmh6r\" (UID: \"0df87f5d-31f4-4571-82b6-fe75c434ff27\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:05.154542 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.154522 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0df87f5d-31f4-4571-82b6-fe75c434ff27-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qmh6r\" (UID: \"0df87f5d-31f4-4571-82b6-fe75c434ff27\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:05.154580 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.154556 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b456f4f-071a-4e5f-b8b6-51c3ee790656-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.154580 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.154573 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3b456f4f-071a-4e5f-b8b6-51c3ee790656-snapshots\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.154643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.154590 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6hlp\" (UniqueName: \"kubernetes.io/projected/3b456f4f-071a-4e5f-b8b6-51c3ee790656-kube-api-access-s6hlp\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.154643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.154605 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-default-certificate\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:05.154643 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.154619 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-stats-auth\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:05.154738 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.154654 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b456f4f-071a-4e5f-b8b6-51c3ee790656-serving-cert\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.154738 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.154679 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f37b227d-fc9c-4a81-85b1-62c04243ad83-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2hhp2\" (UID: \"f37b227d-fc9c-4a81-85b1-62c04243ad83\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2hhp2" Nov 04 03:49:05.154738 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.154695 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b456f4f-071a-4e5f-b8b6-51c3ee790656-service-ca-bundle\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.174602 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.174579 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 03:49:05.174743 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.174728 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 03:49:05.174803 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.174794 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 04 03:49:05.174841 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.174803 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 03:49:05.174915 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.174896 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 03:49:05.174995 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.174952 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 04 03:49:05.174995 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.174959 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 04 03:49:05.175097 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.175078 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 03:49:05.175187 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.175120 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:49:05.175187 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.175135 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 03:49:05.175187 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.175139 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 04 03:49:05.175187 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.175168 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 03:49:05.175187 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.175172 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 03:49:05.175409 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.175394 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 04 03:49:05.175467 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.175430 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:49:05.175467 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.175447 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 03:49:05.175467 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.175457 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:49:05.175612 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.175512 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:49:05.175612 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.175588 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 04 03:49:05.175714 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.175676 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 04 03:49:05.176425 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.176407 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 03:49:05.176506 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.176450 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 04 03:49:05.176558 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.176520 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 04 03:49:05.176558 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.176539 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 03:49:05.176656 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.176563 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 04 03:49:05.176656 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.176572 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:49:05.176890 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.176871 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 04 03:49:05.177513 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.177493 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 04 03:49:05.180422 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.180403 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 04 03:49:05.180529 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.180438 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 04 03:49:05.181624 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.181602 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 04 03:49:05.181684 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.181631 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 04 03:49:05.186584 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.186568 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 04 03:49:05.186664 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.186585 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 04 03:49:05.186664 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.186601 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-75rnz\"" Nov 04 03:49:05.187326 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.187311 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 04 03:49:05.187424 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.187378 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 04 03:49:05.187424 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.187394 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 04 03:49:05.187424 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.187411 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-x27cl\"" Nov 04 03:49:05.187616 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.187437 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-slzqd\"" Nov 04 03:49:05.187616 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.187504 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 04 03:49:05.187616 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.187513 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-c9pqc\"" Nov 04 03:49:05.187616 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.187537 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 04 03:49:05.187616 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.187567 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 04 03:49:05.187616 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.187604 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-b8lr8\"" Nov 04 03:49:05.187851 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.187636 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 04 03:49:05.187851 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.187663 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 04 03:49:05.187851 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.187760 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-k94c2\"" Nov 04 03:49:05.187851 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.187813 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9xsbw\"" Nov 04 03:49:05.189585 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.189347 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 03:49:05.189585 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.189372 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kk77b\"" Nov 04 03:49:05.189585 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.189394 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ccqtd\"" Nov 04 03:49:05.189585 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.189426 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vzqjs\"" Nov 04 03:49:05.189585 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.189432 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 04 03:49:05.189585 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.189440 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 03:49:05.189585 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.189448 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 04 03:49:05.189585 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.189503 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-49k7r\"" Nov 04 03:49:05.255414 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.255392 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjrzz\" (UniqueName: \"kubernetes.io/projected/bcc0d689-48b6-467e-bdd0-6a03878a52b6-kube-api-access-hjrzz\") pod \"ingress-canary-chk4t\" (UID: \"bcc0d689-48b6-467e-bdd0-6a03878a52b6\") " pod="openshift-ingress-canary/ingress-canary-chk4t" Nov 04 03:49:05.255414 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.255415 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-trusted-ca\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.255567 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.255436 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f37b227d-fc9c-4a81-85b1-62c04243ad83-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2hhp2\" (UID: \"f37b227d-fc9c-4a81-85b1-62c04243ad83\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2hhp2" Nov 04 03:49:05.255567 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.255454 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b456f4f-071a-4e5f-b8b6-51c3ee790656-service-ca-bundle\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.255567 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.255505 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcc0d689-48b6-467e-bdd0-6a03878a52b6-cert\") pod \"ingress-canary-chk4t\" (UID: \"bcc0d689-48b6-467e-bdd0-6a03878a52b6\") " pod="openshift-ingress-canary/ingress-canary-chk4t" Nov 04 03:49:05.255567 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.255545 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-image-registry-private-configuration\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.255744 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.255601 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knwrb\" (UniqueName: \"kubernetes.io/projected/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-kube-api-access-knwrb\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:05.255744 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.255663 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 03:49:05.255744 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.255721 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-bound-sa-token\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.255744 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.255736 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f37b227d-fc9c-4a81-85b1-62c04243ad83-networking-console-plugin-cert podName:f37b227d-fc9c-4a81-85b1-62c04243ad83 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:05.755716947 +0000 UTC m=+30.595627445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f37b227d-fc9c-4a81-85b1-62c04243ad83-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2hhp2" (UID: "f37b227d-fc9c-4a81-85b1-62c04243ad83") : secret "networking-console-plugin-cert" not found Nov 04 03:49:05.255886 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.255766 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:05.255886 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.255843 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf4rk\" (UniqueName: \"kubernetes.io/projected/1f0a14d4-9771-4aa2-a6bd-87f5f73ee931-kube-api-access-sf4rk\") pod \"kas-connection-checker-685895c7c9-qq74s\" (UID: \"1f0a14d4-9771-4aa2-a6bd-87f5f73ee931\") " pod="kube-system/kas-connection-checker-685895c7c9-qq74s" Nov 04 03:49:05.255886 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.255876 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-certificates\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.255994 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.255941 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-serving-cert\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:05.255994 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.255973 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f37b227d-fc9c-4a81-85b1-62c04243ad83-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2hhp2\" (UID: \"f37b227d-fc9c-4a81-85b1-62c04243ad83\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2hhp2" Nov 04 03:49:05.255994 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.255987 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b456f4f-071a-4e5f-b8b6-51c3ee790656-service-ca-bundle\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.256083 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256002 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-metrics-certs\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:05.256083 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256027 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-trusted-ca\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.256083 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256058 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0df87f5d-31f4-4571-82b6-fe75c434ff27-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qmh6r\" (UID: \"0df87f5d-31f4-4571-82b6-fe75c434ff27\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:05.256083 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.256068 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 03:49:05.256219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256086 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-config-volume\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:05.256219 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.256115 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-metrics-certs podName:1b81b9bc-4eb5-48a7-98e1-6c7a98117f57 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:05.756102034 +0000 UTC m=+30.596012522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-metrics-certs") pod "router-default-78ccf8f879-s4sx4" (UID: "1b81b9bc-4eb5-48a7-98e1-6c7a98117f57") : secret "router-metrics-certs-default" not found Nov 04 03:49:05.256219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256130 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szjzl\" (UniqueName: \"kubernetes.io/projected/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-kube-api-access-szjzl\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:05.256219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256155 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2052e7f-5ee8-4066-bb29-cd227fa0965b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm\" (UID: \"d2052e7f-5ee8-4066-bb29-cd227fa0965b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" Nov 04 03:49:05.256219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256172 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5t7v\" (UniqueName: \"kubernetes.io/projected/30f18911-ab5a-44e3-88b2-6508325ceb39-kube-api-access-q5t7v\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:05.256219 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.256206 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 03:49:05.256219 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256210 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-tmp-dir\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:05.256513 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.256278 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0df87f5d-31f4-4571-82b6-fe75c434ff27-cluster-monitoring-operator-tls podName:0df87f5d-31f4-4571-82b6-fe75c434ff27 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:05.756262667 +0000 UTC m=+30.596173164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0df87f5d-31f4-4571-82b6-fe75c434ff27-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qmh6r" (UID: "0df87f5d-31f4-4571-82b6-fe75c434ff27") : secret "cluster-monitoring-operator-tls" not found Nov 04 03:49:05.256513 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256319 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.256513 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256347 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjgp4\" (UniqueName: \"kubernetes.io/projected/3fbef9e3-d5de-4a35-8745-2b1dc357c52b-kube-api-access-rjgp4\") pod \"volume-data-source-validator-58c9f5f46d-vjhfk\" (UID: \"3fbef9e3-d5de-4a35-8745-2b1dc357c52b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vjhfk" Nov 04 03:49:05.256513 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256422 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcjkb\" (UniqueName: \"kubernetes.io/projected/d2052e7f-5ee8-4066-bb29-cd227fa0965b-kube-api-access-pcjkb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm\" (UID: \"d2052e7f-5ee8-4066-bb29-cd227fa0965b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" Nov 04 03:49:05.256513 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256454 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvg2z\" (UniqueName: \"kubernetes.io/projected/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-kube-api-access-zvg2z\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:05.256513 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256481 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-installation-pull-secrets\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.256513 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256509 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhvtl\" (UniqueName: \"kubernetes.io/projected/1c2ac1f5-6e08-48e3-ac83-a55f9181f4be-kube-api-access-zhvtl\") pod \"kas-connection-checker-685895c7c9-wc5zp\" (UID: \"1c2ac1f5-6e08-48e3-ac83-a55f9181f4be\") " pod="kube-system/kas-connection-checker-685895c7c9-wc5zp" Nov 04 03:49:05.256803 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256578 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f37b227d-fc9c-4a81-85b1-62c04243ad83-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2hhp2\" (UID: \"f37b227d-fc9c-4a81-85b1-62c04243ad83\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2hhp2" Nov 04 03:49:05.256803 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256601 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b456f4f-071a-4e5f-b8b6-51c3ee790656-serving-cert\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.256803 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256634 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-trusted-ca\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:05.256803 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256650 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-image-registry-private-configuration\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.256803 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256679 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d313ca1-e640-42d9-a9f7-5e3e8918dc77-config\") pod \"service-ca-operator-544d749c8c-2jgg8\" (UID: \"3d313ca1-e640-42d9-a9f7-5e3e8918dc77\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" Nov 04 03:49:05.256803 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256697 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b456f4f-071a-4e5f-b8b6-51c3ee790656-tmp\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.256803 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256722 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.256803 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256796 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/363d8742-c0a7-405c-9683-7b12afbd8bcc-ca-trust-extracted\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.257166 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256827 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-952rw\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-kube-api-access-952rw\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.257166 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256894 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr88g\" (UniqueName: \"kubernetes.io/projected/27d72728-5623-41e0-bee5-a09937f9f87a-kube-api-access-kr88g\") pod \"kas-connection-checker-685895c7c9-8v2sd\" (UID: \"27d72728-5623-41e0-bee5-a09937f9f87a\") " pod="kube-system/kas-connection-checker-685895c7c9-8v2sd" Nov 04 03:49:05.257166 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.256982 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b456f4f-071a-4e5f-b8b6-51c3ee790656-tmp\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.257166 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257008 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9fvw\" (UniqueName: \"kubernetes.io/projected/0df87f5d-31f4-4571-82b6-fe75c434ff27-kube-api-access-w9fvw\") pod \"cluster-monitoring-operator-884576c57-qmh6r\" (UID: \"0df87f5d-31f4-4571-82b6-fe75c434ff27\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:05.257166 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257032 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:05.257166 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257049 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:05.257166 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257077 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rpkk\" (UniqueName: \"kubernetes.io/projected/21ac7748-5e3d-427e-bd15-93cf4754cb0e-kube-api-access-5rpkk\") pod \"network-check-source-6f78d59f99-5xh74\" (UID: \"21ac7748-5e3d-427e-bd15-93cf4754cb0e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5xh74" Nov 04 03:49:05.257166 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.257149 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle podName:1b81b9bc-4eb5-48a7-98e1-6c7a98117f57 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:05.757137178 +0000 UTC m=+30.597047663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle") pod "router-default-78ccf8f879-s4sx4" (UID: "1b81b9bc-4eb5-48a7-98e1-6c7a98117f57") : configmap references non-existent config key: service-ca.crt Nov 04 03:49:05.257534 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257187 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d2052e7f-5ee8-4066-bb29-cd227fa0965b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm\" (UID: \"d2052e7f-5ee8-4066-bb29-cd227fa0965b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" Nov 04 03:49:05.257534 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257233 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:05.257534 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257273 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0df87f5d-31f4-4571-82b6-fe75c434ff27-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qmh6r\" (UID: \"0df87f5d-31f4-4571-82b6-fe75c434ff27\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:05.257534 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257300 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-default-certificate\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:05.257534 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257323 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-stats-auth\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:05.257534 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257357 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ce83f95-758c-4fd2-9678-441fe4309da7-ca-trust-extracted\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.257534 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257384 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-installation-pull-secrets\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.257534 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257522 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqp55\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-kube-api-access-sqp55\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.257973 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257586 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b456f4f-071a-4e5f-b8b6-51c3ee790656-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.257973 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257618 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw88q\" (UniqueName: \"kubernetes.io/projected/3d313ca1-e640-42d9-a9f7-5e3e8918dc77-kube-api-access-rw88q\") pod \"service-ca-operator-544d749c8c-2jgg8\" (UID: \"3d313ca1-e640-42d9-a9f7-5e3e8918dc77\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" Nov 04 03:49:05.257973 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257649 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3b456f4f-071a-4e5f-b8b6-51c3ee790656-snapshots\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.257973 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257671 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-bound-sa-token\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.257973 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257705 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d313ca1-e640-42d9-a9f7-5e3e8918dc77-serving-cert\") pod \"service-ca-operator-544d749c8c-2jgg8\" (UID: \"3d313ca1-e640-42d9-a9f7-5e3e8918dc77\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" Nov 04 03:49:05.257973 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257731 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6hlp\" (UniqueName: \"kubernetes.io/projected/3b456f4f-071a-4e5f-b8b6-51c3ee790656-kube-api-access-s6hlp\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.257973 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257754 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-certificates\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.257973 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.257778 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-config\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:05.258355 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.258013 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0df87f5d-31f4-4571-82b6-fe75c434ff27-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qmh6r\" (UID: \"0df87f5d-31f4-4571-82b6-fe75c434ff27\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:05.258429 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.258404 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b456f4f-071a-4e5f-b8b6-51c3ee790656-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.258487 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.258430 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3b456f4f-071a-4e5f-b8b6-51c3ee790656-snapshots\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.260134 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.260108 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b456f4f-071a-4e5f-b8b6-51c3ee790656-serving-cert\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.260316 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.260293 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-default-certificate\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:05.260451 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.260312 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-stats-auth\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:05.268764 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.268740 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf4rk\" (UniqueName: \"kubernetes.io/projected/1f0a14d4-9771-4aa2-a6bd-87f5f73ee931-kube-api-access-sf4rk\") pod \"kas-connection-checker-685895c7c9-qq74s\" (UID: \"1f0a14d4-9771-4aa2-a6bd-87f5f73ee931\") " pod="kube-system/kas-connection-checker-685895c7c9-qq74s" Nov 04 03:49:05.268874 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.268780 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhvtl\" (UniqueName: \"kubernetes.io/projected/1c2ac1f5-6e08-48e3-ac83-a55f9181f4be-kube-api-access-zhvtl\") pod \"kas-connection-checker-685895c7c9-wc5zp\" (UID: \"1c2ac1f5-6e08-48e3-ac83-a55f9181f4be\") " pod="kube-system/kas-connection-checker-685895c7c9-wc5zp" Nov 04 03:49:05.269171 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.269149 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knwrb\" (UniqueName: \"kubernetes.io/projected/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-kube-api-access-knwrb\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:05.270623 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.270599 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9fvw\" (UniqueName: \"kubernetes.io/projected/0df87f5d-31f4-4571-82b6-fe75c434ff27-kube-api-access-w9fvw\") pod \"cluster-monitoring-operator-884576c57-qmh6r\" (UID: \"0df87f5d-31f4-4571-82b6-fe75c434ff27\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:05.270716 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.270610 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr88g\" (UniqueName: \"kubernetes.io/projected/27d72728-5623-41e0-bee5-a09937f9f87a-kube-api-access-kr88g\") pod \"kas-connection-checker-685895c7c9-8v2sd\" (UID: \"27d72728-5623-41e0-bee5-a09937f9f87a\") " pod="kube-system/kas-connection-checker-685895c7c9-8v2sd" Nov 04 03:49:05.270784 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.270724 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6hlp\" (UniqueName: \"kubernetes.io/projected/3b456f4f-071a-4e5f-b8b6-51c3ee790656-kube-api-access-s6hlp\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.358075 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358045 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-tmp-dir\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:05.358075 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358076 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.358252 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358094 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjgp4\" (UniqueName: \"kubernetes.io/projected/3fbef9e3-d5de-4a35-8745-2b1dc357c52b-kube-api-access-rjgp4\") pod \"volume-data-source-validator-58c9f5f46d-vjhfk\" (UID: \"3fbef9e3-d5de-4a35-8745-2b1dc357c52b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vjhfk" Nov 04 03:49:05.358252 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358121 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcjkb\" (UniqueName: \"kubernetes.io/projected/d2052e7f-5ee8-4066-bb29-cd227fa0965b-kube-api-access-pcjkb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm\" (UID: \"d2052e7f-5ee8-4066-bb29-cd227fa0965b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" Nov 04 03:49:05.358252 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358138 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvg2z\" (UniqueName: \"kubernetes.io/projected/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-kube-api-access-zvg2z\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:05.358488 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.358232 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:49:05.358488 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.358267 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8648b894c6-t5rf2: secret "image-registry-tls" not found Nov 04 03:49:05.358488 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358272 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-installation-pull-secrets\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.358488 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.358333 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls podName:3ce83f95-758c-4fd2-9678-441fe4309da7 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:05.858304974 +0000 UTC m=+30.698215467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls") pod "image-registry-8648b894c6-t5rf2" (UID: "3ce83f95-758c-4fd2-9678-441fe4309da7") : secret "image-registry-tls" not found Nov 04 03:49:05.358488 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358355 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-trusted-ca\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:05.358488 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358400 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-image-registry-private-configuration\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.358488 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358356 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-tmp-dir\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:05.358488 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358450 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d313ca1-e640-42d9-a9f7-5e3e8918dc77-config\") pod \"service-ca-operator-544d749c8c-2jgg8\" (UID: \"3d313ca1-e640-42d9-a9f7-5e3e8918dc77\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" Nov 04 03:49:05.358488 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358470 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.358488 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358486 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/363d8742-c0a7-405c-9683-7b12afbd8bcc-ca-trust-extracted\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.359035 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358509 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-952rw\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-kube-api-access-952rw\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.359035 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358554 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:05.359035 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.358571 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:49:05.359035 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358579 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rpkk\" (UniqueName: \"kubernetes.io/projected/21ac7748-5e3d-427e-bd15-93cf4754cb0e-kube-api-access-5rpkk\") pod \"network-check-source-6f78d59f99-5xh74\" (UID: \"21ac7748-5e3d-427e-bd15-93cf4754cb0e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5xh74" Nov 04 03:49:05.359035 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.358586 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6448ff747c-dlpnw: secret "image-registry-tls" not found Nov 04 03:49:05.359035 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358607 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d2052e7f-5ee8-4066-bb29-cd227fa0965b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm\" (UID: \"d2052e7f-5ee8-4066-bb29-cd227fa0965b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" Nov 04 03:49:05.359035 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.358631 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls podName:363d8742-c0a7-405c-9683-7b12afbd8bcc nodeName:}" failed. No retries permitted until 2026-11-04 03:49:05.858617577 +0000 UTC m=+30.698528080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls") pod "image-registry-6448ff747c-dlpnw" (UID: "363d8742-c0a7-405c-9683-7b12afbd8bcc") : secret "image-registry-tls" not found Nov 04 03:49:05.359035 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358650 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:05.359035 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358696 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ce83f95-758c-4fd2-9678-441fe4309da7-ca-trust-extracted\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.359035 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358723 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-installation-pull-secrets\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.359035 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358755 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqp55\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-kube-api-access-sqp55\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.359035 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358841 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw88q\" (UniqueName: \"kubernetes.io/projected/3d313ca1-e640-42d9-a9f7-5e3e8918dc77-kube-api-access-rw88q\") pod \"service-ca-operator-544d749c8c-2jgg8\" (UID: \"3d313ca1-e640-42d9-a9f7-5e3e8918dc77\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" Nov 04 03:49:05.359035 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358868 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-bound-sa-token\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.359035 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358894 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/363d8742-c0a7-405c-9683-7b12afbd8bcc-ca-trust-extracted\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.359035 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358939 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d313ca1-e640-42d9-a9f7-5e3e8918dc77-serving-cert\") pod \"service-ca-operator-544d749c8c-2jgg8\" (UID: \"3d313ca1-e640-42d9-a9f7-5e3e8918dc77\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" Nov 04 03:49:05.359035 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.358973 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-certificates\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.359974 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.359003 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-config\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:05.359974 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.359027 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 03:49:05.359974 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.359037 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjrzz\" (UniqueName: \"kubernetes.io/projected/bcc0d689-48b6-467e-bdd0-6a03878a52b6-kube-api-access-hjrzz\") pod \"ingress-canary-chk4t\" (UID: \"bcc0d689-48b6-467e-bdd0-6a03878a52b6\") " pod="openshift-ingress-canary/ingress-canary-chk4t" Nov 04 03:49:05.359974 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.359063 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-trusted-ca\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.359974 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.359076 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls podName:30f18911-ab5a-44e3-88b2-6508325ceb39 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:05.859059755 +0000 UTC m=+30.698970259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6x2fz" (UID: "30f18911-ab5a-44e3-88b2-6508325ceb39") : secret "samples-operator-tls" not found Nov 04 03:49:05.359974 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.359096 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d313ca1-e640-42d9-a9f7-5e3e8918dc77-config\") pod \"service-ca-operator-544d749c8c-2jgg8\" (UID: \"3d313ca1-e640-42d9-a9f7-5e3e8918dc77\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" Nov 04 03:49:05.359974 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.359147 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcc0d689-48b6-467e-bdd0-6a03878a52b6-cert\") pod \"ingress-canary-chk4t\" (UID: \"bcc0d689-48b6-467e-bdd0-6a03878a52b6\") " pod="openshift-ingress-canary/ingress-canary-chk4t" Nov 04 03:49:05.359974 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.359174 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-image-registry-private-configuration\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.359974 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.359206 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-bound-sa-token\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.359974 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.359230 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:05.359974 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.359264 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-trusted-ca\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:05.359974 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.359266 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-certificates\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.359974 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.359343 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-serving-cert\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:05.359974 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.359568 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-trusted-ca\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.359974 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.359624 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-config-volume\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:05.359974 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.359654 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szjzl\" (UniqueName: \"kubernetes.io/projected/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-kube-api-access-szjzl\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:05.360977 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.359689 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2052e7f-5ee8-4066-bb29-cd227fa0965b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm\" (UID: \"d2052e7f-5ee8-4066-bb29-cd227fa0965b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" Nov 04 03:49:05.360977 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.359719 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5t7v\" (UniqueName: \"kubernetes.io/projected/30f18911-ab5a-44e3-88b2-6508325ceb39-kube-api-access-q5t7v\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:05.360977 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.359805 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-certificates\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.360977 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.359949 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-certificates\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.360977 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.360216 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-config\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:05.360977 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.360298 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ce83f95-758c-4fd2-9678-441fe4309da7-ca-trust-extracted\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.360977 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.359229 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d2052e7f-5ee8-4066-bb29-cd227fa0965b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm\" (UID: \"d2052e7f-5ee8-4066-bb29-cd227fa0965b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" Nov 04 03:49:05.361412 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.360979 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:05.361412 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.361028 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-trusted-ca\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.361412 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.361090 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 03:49:05.361412 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.361134 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls podName:7b632ca9-a695-450e-a1a9-b5357ebf4d7d nodeName:}" failed. No retries permitted until 2026-11-04 03:49:05.861120704 +0000 UTC m=+30.701031206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls") pod "dns-default-pvrd4" (UID: "7b632ca9-a695-450e-a1a9-b5357ebf4d7d") : secret "dns-default-metrics-tls" not found Nov 04 03:49:05.361412 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.361147 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-trusted-ca\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.361412 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.361199 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-config-volume\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:05.361745 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.361577 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-image-registry-private-configuration\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.362151 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.362102 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-installation-pull-secrets\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.363260 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.363241 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-serving-cert\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:05.363358 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.363340 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2052e7f-5ee8-4066-bb29-cd227fa0965b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm\" (UID: \"d2052e7f-5ee8-4066-bb29-cd227fa0965b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" Nov 04 03:49:05.363458 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.363412 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-installation-pull-secrets\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.363671 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.363655 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d313ca1-e640-42d9-a9f7-5e3e8918dc77-serving-cert\") pod \"service-ca-operator-544d749c8c-2jgg8\" (UID: \"3d313ca1-e640-42d9-a9f7-5e3e8918dc77\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" Nov 04 03:49:05.364025 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.364005 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-image-registry-private-configuration\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.365018 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.364998 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcc0d689-48b6-467e-bdd0-6a03878a52b6-cert\") pod \"ingress-canary-chk4t\" (UID: \"bcc0d689-48b6-467e-bdd0-6a03878a52b6\") " pod="openshift-ingress-canary/ingress-canary-chk4t" Nov 04 03:49:05.372541 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.372521 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjgp4\" (UniqueName: \"kubernetes.io/projected/3fbef9e3-d5de-4a35-8745-2b1dc357c52b-kube-api-access-rjgp4\") pod \"volume-data-source-validator-58c9f5f46d-vjhfk\" (UID: \"3fbef9e3-d5de-4a35-8745-2b1dc357c52b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vjhfk" Nov 04 03:49:05.372725 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.372709 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcjkb\" (UniqueName: \"kubernetes.io/projected/d2052e7f-5ee8-4066-bb29-cd227fa0965b-kube-api-access-pcjkb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm\" (UID: \"d2052e7f-5ee8-4066-bb29-cd227fa0965b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" Nov 04 03:49:05.375486 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.375460 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-952rw\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-kube-api-access-952rw\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.375693 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.375676 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjrzz\" (UniqueName: \"kubernetes.io/projected/bcc0d689-48b6-467e-bdd0-6a03878a52b6-kube-api-access-hjrzz\") pod \"ingress-canary-chk4t\" (UID: \"bcc0d689-48b6-467e-bdd0-6a03878a52b6\") " pod="openshift-ingress-canary/ingress-canary-chk4t" Nov 04 03:49:05.376271 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.376249 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szjzl\" (UniqueName: \"kubernetes.io/projected/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-kube-api-access-szjzl\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:05.376524 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.376500 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-bound-sa-token\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.376935 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.376903 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rpkk\" (UniqueName: \"kubernetes.io/projected/21ac7748-5e3d-427e-bd15-93cf4754cb0e-kube-api-access-5rpkk\") pod \"network-check-source-6f78d59f99-5xh74\" (UID: \"21ac7748-5e3d-427e-bd15-93cf4754cb0e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5xh74" Nov 04 03:49:05.378082 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.378051 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqp55\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-kube-api-access-sqp55\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.378541 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.378524 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-bound-sa-token\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.379935 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.379906 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qq74s" Nov 04 03:49:05.380426 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.380406 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw88q\" (UniqueName: \"kubernetes.io/projected/3d313ca1-e640-42d9-a9f7-5e3e8918dc77-kube-api-access-rw88q\") pod \"service-ca-operator-544d749c8c-2jgg8\" (UID: \"3d313ca1-e640-42d9-a9f7-5e3e8918dc77\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" Nov 04 03:49:05.380828 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.380810 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5t7v\" (UniqueName: \"kubernetes.io/projected/30f18911-ab5a-44e3-88b2-6508325ceb39-kube-api-access-q5t7v\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:05.381582 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.381560 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvg2z\" (UniqueName: \"kubernetes.io/projected/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-kube-api-access-zvg2z\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:05.396173 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.396160 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8v2sd" Nov 04 03:49:05.414550 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.414530 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wc5zp" Nov 04 03:49:05.442903 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.442742 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:05.443786 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.443762 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" Nov 04 03:49:05.459044 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.459012 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:05.479322 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.479291 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" Nov 04 03:49:05.501440 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.499841 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vjhfk" Nov 04 03:49:05.525571 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.525062 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5xh74" Nov 04 03:49:05.554876 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.553218 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-chk4t" Nov 04 03:49:05.719486 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.719461 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wc5zp"] Nov 04 03:49:05.722321 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:49:05.722297 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2ac1f5_6e08_48e3_ac83_a55f9181f4be.slice/crio-ea290fb0d10aeea5436aeedfbcd14d5e027d56b395c4f74388f7269f1b8542cc WatchSource:0}: Error finding container ea290fb0d10aeea5436aeedfbcd14d5e027d56b395c4f74388f7269f1b8542cc: Status 404 returned error can't find the container with id ea290fb0d10aeea5436aeedfbcd14d5e027d56b395c4f74388f7269f1b8542cc Nov 04 03:49:05.722399 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.722334 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qq74s"] Nov 04 03:49:05.723559 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.723534 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8v2sd"] Nov 04 03:49:05.725144 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:49:05.725121 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0a14d4_9771_4aa2_a6bd_87f5f73ee931.slice/crio-97c7fe96419bb1a3c65f7184da334ca0636c3182e80f416470a8a21bf6c41b61 WatchSource:0}: Error finding container 97c7fe96419bb1a3c65f7184da334ca0636c3182e80f416470a8a21bf6c41b61: Status 404 returned error can't find the container with id 97c7fe96419bb1a3c65f7184da334ca0636c3182e80f416470a8a21bf6c41b61 Nov 04 03:49:05.726433 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:49:05.726408 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d72728_5623_41e0_bee5_a09937f9f87a.slice/crio-053490f48702475ba5ceeba97610c47fbb3bd28090536d601c3ca63e156b8b34 WatchSource:0}: Error finding container 053490f48702475ba5ceeba97610c47fbb3bd28090536d601c3ca63e156b8b34: Status 404 returned error can't find the container with id 053490f48702475ba5ceeba97610c47fbb3bd28090536d601c3ca63e156b8b34 Nov 04 03:49:05.763472 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.762952 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:05.763472 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.763025 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f37b227d-fc9c-4a81-85b1-62c04243ad83-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2hhp2\" (UID: \"f37b227d-fc9c-4a81-85b1-62c04243ad83\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2hhp2" Nov 04 03:49:05.763472 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.763085 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-metrics-certs\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:05.763472 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.763115 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0df87f5d-31f4-4571-82b6-fe75c434ff27-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qmh6r\" (UID: \"0df87f5d-31f4-4571-82b6-fe75c434ff27\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:05.763472 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.763143 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle podName:1b81b9bc-4eb5-48a7-98e1-6c7a98117f57 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:06.763125144 +0000 UTC m=+31.603035646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle") pod "router-default-78ccf8f879-s4sx4" (UID: "1b81b9bc-4eb5-48a7-98e1-6c7a98117f57") : configmap references non-existent config key: service-ca.crt Nov 04 03:49:05.763472 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.763189 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 03:49:05.763472 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.763228 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0df87f5d-31f4-4571-82b6-fe75c434ff27-cluster-monitoring-operator-tls podName:0df87f5d-31f4-4571-82b6-fe75c434ff27 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:06.763216171 +0000 UTC m=+31.603126658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0df87f5d-31f4-4571-82b6-fe75c434ff27-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qmh6r" (UID: "0df87f5d-31f4-4571-82b6-fe75c434ff27") : secret "cluster-monitoring-operator-tls" not found Nov 04 03:49:05.763472 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.763327 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 03:49:05.763472 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.763360 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f37b227d-fc9c-4a81-85b1-62c04243ad83-networking-console-plugin-cert podName:f37b227d-fc9c-4a81-85b1-62c04243ad83 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:06.763349545 +0000 UTC m=+31.603260032 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f37b227d-fc9c-4a81-85b1-62c04243ad83-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2hhp2" (UID: "f37b227d-fc9c-4a81-85b1-62c04243ad83") : secret "networking-console-plugin-cert" not found Nov 04 03:49:05.763472 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.763420 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 03:49:05.763472 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.763449 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-metrics-certs podName:1b81b9bc-4eb5-48a7-98e1-6c7a98117f57 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:06.76343873 +0000 UTC m=+31.603349215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-metrics-certs") pod "router-default-78ccf8f879-s4sx4" (UID: "1b81b9bc-4eb5-48a7-98e1-6c7a98117f57") : secret "router-metrics-certs-default" not found Nov 04 03:49:05.785365 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.785301 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-gm5s7"] Nov 04 03:49:05.796219 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:49:05.796178 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b456f4f_071a_4e5f_b8b6_51c3ee790656.slice/crio-e47ff45ec5397b88dafbf3a2734ad5b34db3d4d6e13746a247bb5c3e9cae99e3 WatchSource:0}: Error finding container e47ff45ec5397b88dafbf3a2734ad5b34db3d4d6e13746a247bb5c3e9cae99e3: Status 404 returned error can't find the container with id e47ff45ec5397b88dafbf3a2734ad5b34db3d4d6e13746a247bb5c3e9cae99e3 Nov 04 03:49:05.802681 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.802383 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8"] Nov 04 03:49:05.806914 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:49:05.806887 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d313ca1_e640_42d9_a9f7_5e3e8918dc77.slice/crio-c56d598b430b5f4e609403bdd5c77a25b76bb388cabbe0671da260a43c084ec8 WatchSource:0}: Error finding container c56d598b430b5f4e609403bdd5c77a25b76bb388cabbe0671da260a43c084ec8: Status 404 returned error can't find the container with id c56d598b430b5f4e609403bdd5c77a25b76bb388cabbe0671da260a43c084ec8 Nov 04 03:49:05.831199 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.831170 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wc5zp" event={"ID":"1c2ac1f5-6e08-48e3-ac83-a55f9181f4be","Type":"ContainerStarted","Data":"ea290fb0d10aeea5436aeedfbcd14d5e027d56b395c4f74388f7269f1b8542cc"} Nov 04 03:49:05.832379 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.832328 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-gm5s7" event={"ID":"3b456f4f-071a-4e5f-b8b6-51c3ee790656","Type":"ContainerStarted","Data":"e47ff45ec5397b88dafbf3a2734ad5b34db3d4d6e13746a247bb5c3e9cae99e3"} Nov 04 03:49:05.833629 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.833605 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qq74s" event={"ID":"1f0a14d4-9771-4aa2-a6bd-87f5f73ee931","Type":"ContainerStarted","Data":"97c7fe96419bb1a3c65f7184da334ca0636c3182e80f416470a8a21bf6c41b61"} Nov 04 03:49:05.834811 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.834787 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" event={"ID":"3d313ca1-e640-42d9-a9f7-5e3e8918dc77","Type":"ContainerStarted","Data":"c56d598b430b5f4e609403bdd5c77a25b76bb388cabbe0671da260a43c084ec8"} Nov 04 03:49:05.836889 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.836862 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8v2sd" event={"ID":"27d72728-5623-41e0-bee5-a09937f9f87a","Type":"ContainerStarted","Data":"053490f48702475ba5ceeba97610c47fbb3bd28090536d601c3ca63e156b8b34"} Nov 04 03:49:05.844449 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.844398 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm"] Nov 04 03:49:05.849522 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:49:05.849502 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2052e7f_5ee8_4066_bb29_cd227fa0965b.slice/crio-cd27c2ebd27cf211c4d38ad9efc9a443a46483c9bc3aec8bdc605979b7ab4467 WatchSource:0}: Error finding container cd27c2ebd27cf211c4d38ad9efc9a443a46483c9bc3aec8bdc605979b7ab4467: Status 404 returned error can't find the container with id cd27c2ebd27cf211c4d38ad9efc9a443a46483c9bc3aec8bdc605979b7ab4467 Nov 04 03:49:05.850755 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.850694 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vjhfk"] Nov 04 03:49:05.853825 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.853811 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rr5bv"] Nov 04 03:49:05.859791 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:49:05.859754 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbef9e3_d5de_4a35_8745_2b1dc357c52b.slice/crio-75dd49f74d6e64dd95776fe7d6053b7f8010387a3ac35b7c7413713dbc41ba5e WatchSource:0}: Error finding container 75dd49f74d6e64dd95776fe7d6053b7f8010387a3ac35b7c7413713dbc41ba5e: Status 404 returned error can't find the container with id 75dd49f74d6e64dd95776fe7d6053b7f8010387a3ac35b7c7413713dbc41ba5e Nov 04 03:49:05.863770 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.863667 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:05.864058 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.863960 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 03:49:05.864792 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.864236 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:49:05.864792 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.864254 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8648b894c6-t5rf2: secret "image-registry-tls" not found Nov 04 03:49:05.864792 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.864282 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls podName:7b632ca9-a695-450e-a1a9-b5357ebf4d7d nodeName:}" failed. No retries permitted until 2026-11-04 03:49:06.864256128 +0000 UTC m=+31.704166637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls") pod "dns-default-pvrd4" (UID: "7b632ca9-a695-450e-a1a9-b5357ebf4d7d") : secret "dns-default-metrics-tls" not found Nov 04 03:49:05.864792 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.864144 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:05.864792 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.864320 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls podName:3ce83f95-758c-4fd2-9678-441fe4309da7 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:06.864295303 +0000 UTC m=+31.704205794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls") pod "image-registry-8648b894c6-t5rf2" (UID: "3ce83f95-758c-4fd2-9678-441fe4309da7") : secret "image-registry-tls" not found Nov 04 03:49:05.864792 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.864430 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:05.864792 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.864485 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:05.864792 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.864559 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:49:05.864792 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.864571 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6448ff747c-dlpnw: secret "image-registry-tls" not found Nov 04 03:49:05.864792 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.864614 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls podName:363d8742-c0a7-405c-9683-7b12afbd8bcc nodeName:}" failed. No retries permitted until 2026-11-04 03:49:06.864598515 +0000 UTC m=+31.704509013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls") pod "image-registry-6448ff747c-dlpnw" (UID: "363d8742-c0a7-405c-9683-7b12afbd8bcc") : secret "image-registry-tls" not found Nov 04 03:49:05.865367 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.865312 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 03:49:05.865367 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:05.865352 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls podName:30f18911-ab5a-44e3-88b2-6508325ceb39 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:06.865341322 +0000 UTC m=+31.705251826 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6x2fz" (UID: "30f18911-ab5a-44e3-88b2-6508325ceb39") : secret "samples-operator-tls" not found Nov 04 03:49:05.907362 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.907342 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-5xh74"] Nov 04 03:49:05.908993 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:49:05.908967 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ac7748_5e3d_427e_bd15_93cf4754cb0e.slice/crio-8ddf93747976c4469160b51e7421723397597f92296ee22a6001853599bd527b WatchSource:0}: Error finding container 8ddf93747976c4469160b51e7421723397597f92296ee22a6001853599bd527b: Status 404 returned error can't find the container with id 8ddf93747976c4469160b51e7421723397597f92296ee22a6001853599bd527b Nov 04 03:49:05.910825 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:05.910805 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-chk4t"] Nov 04 03:49:05.913839 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:49:05.913815 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc0d689_48b6_467e_bdd0_6a03878a52b6.slice/crio-571da96cb8b6a018b0047426b1c5561018ac9f326cf4ad56f325132b4fcbec4b WatchSource:0}: Error finding container 571da96cb8b6a018b0047426b1c5561018ac9f326cf4ad56f325132b4fcbec4b: Status 404 returned error can't find the container with id 571da96cb8b6a018b0047426b1c5561018ac9f326cf4ad56f325132b4fcbec4b Nov 04 03:49:06.592581 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.592546 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:49:06.593145 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.593126 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:49:06.593540 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.593522 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:49:06.598145 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.597872 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-p2j7s\"" Nov 04 03:49:06.598145 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.597978 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 03:49:06.598145 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.598008 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 03:49:06.598145 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.598015 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fmfbg\"" Nov 04 03:49:06.598145 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.598105 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-49g8t\"" Nov 04 03:49:06.774589 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.774553 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:06.775061 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.774638 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f37b227d-fc9c-4a81-85b1-62c04243ad83-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2hhp2\" (UID: \"f37b227d-fc9c-4a81-85b1-62c04243ad83\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2hhp2" Nov 04 03:49:06.775061 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.774718 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-metrics-certs\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:06.775061 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.774754 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0df87f5d-31f4-4571-82b6-fe75c434ff27-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qmh6r\" (UID: \"0df87f5d-31f4-4571-82b6-fe75c434ff27\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:06.775061 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:06.774904 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 03:49:06.775061 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:06.774984 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0df87f5d-31f4-4571-82b6-fe75c434ff27-cluster-monitoring-operator-tls podName:0df87f5d-31f4-4571-82b6-fe75c434ff27 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:08.774966184 +0000 UTC m=+33.614876684 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0df87f5d-31f4-4571-82b6-fe75c434ff27-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qmh6r" (UID: "0df87f5d-31f4-4571-82b6-fe75c434ff27") : secret "cluster-monitoring-operator-tls" not found Nov 04 03:49:06.775422 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:06.775400 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle podName:1b81b9bc-4eb5-48a7-98e1-6c7a98117f57 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:08.775383443 +0000 UTC m=+33.615293937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle") pod "router-default-78ccf8f879-s4sx4" (UID: "1b81b9bc-4eb5-48a7-98e1-6c7a98117f57") : configmap references non-existent config key: service-ca.crt Nov 04 03:49:06.775538 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:06.775482 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 03:49:06.775538 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:06.775518 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f37b227d-fc9c-4a81-85b1-62c04243ad83-networking-console-plugin-cert podName:f37b227d-fc9c-4a81-85b1-62c04243ad83 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:08.775506824 +0000 UTC m=+33.615417314 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f37b227d-fc9c-4a81-85b1-62c04243ad83-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2hhp2" (UID: "f37b227d-fc9c-4a81-85b1-62c04243ad83") : secret "networking-console-plugin-cert" not found Nov 04 03:49:06.775985 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:06.775876 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 03:49:06.775985 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:06.775931 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-metrics-certs podName:1b81b9bc-4eb5-48a7-98e1-6c7a98117f57 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:08.775904249 +0000 UTC m=+33.615814739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-metrics-certs") pod "router-default-78ccf8f879-s4sx4" (UID: "1b81b9bc-4eb5-48a7-98e1-6c7a98117f57") : secret "router-metrics-certs-default" not found Nov 04 03:49:06.857989 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.857672 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qq74s" event={"ID":"1f0a14d4-9771-4aa2-a6bd-87f5f73ee931","Type":"ContainerStarted","Data":"c4b32def4844a7f4eedcee8d147352f79788af5733af5bfa16726232ec272788"} Nov 04 03:49:06.863465 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.863400 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8v2sd" event={"ID":"27d72728-5623-41e0-bee5-a09937f9f87a","Type":"ContainerStarted","Data":"6509f5f179a4cc50f89ef5d015bef28224e49fb31193eabb992f338acfa7dccc"} Nov 04 03:49:06.866989 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.866885 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" event={"ID":"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6","Type":"ContainerStarted","Data":"26e9731be30c71c75b18565bad3d98e6c1838b88918eb12c8f0bb754feca305a"} Nov 04 03:49:06.878792 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.876013 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:06.878792 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.876110 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:06.878792 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.876175 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:06.878792 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.876218 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:06.878792 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:06.876359 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 03:49:06.878792 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:06.876418 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls podName:30f18911-ab5a-44e3-88b2-6508325ceb39 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:08.87640126 +0000 UTC m=+33.716311745 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6x2fz" (UID: "30f18911-ab5a-44e3-88b2-6508325ceb39") : secret "samples-operator-tls" not found Nov 04 03:49:06.878792 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:06.876780 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 03:49:06.878792 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:06.876831 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls podName:7b632ca9-a695-450e-a1a9-b5357ebf4d7d nodeName:}" failed. No retries permitted until 2026-11-04 03:49:08.876814484 +0000 UTC m=+33.716724970 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls") pod "dns-default-pvrd4" (UID: "7b632ca9-a695-450e-a1a9-b5357ebf4d7d") : secret "dns-default-metrics-tls" not found Nov 04 03:49:06.878792 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:06.876908 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:49:06.879460 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:06.879338 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8648b894c6-t5rf2: secret "image-registry-tls" not found Nov 04 03:49:06.879460 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:06.879394 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls podName:3ce83f95-758c-4fd2-9678-441fe4309da7 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:08.879379141 +0000 UTC m=+33.719289634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls") pod "image-registry-8648b894c6-t5rf2" (UID: "3ce83f95-758c-4fd2-9678-441fe4309da7") : secret "image-registry-tls" not found Nov 04 03:49:06.879460 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:06.878682 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:49:06.879460 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:06.879414 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6448ff747c-dlpnw: secret "image-registry-tls" not found Nov 04 03:49:06.879460 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:06.879444 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls podName:363d8742-c0a7-405c-9683-7b12afbd8bcc nodeName:}" failed. No retries permitted until 2026-11-04 03:49:08.879434357 +0000 UTC m=+33.719344849 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls") pod "image-registry-6448ff747c-dlpnw" (UID: "363d8742-c0a7-405c-9683-7b12afbd8bcc") : secret "image-registry-tls" not found Nov 04 03:49:06.881095 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.881035 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-chk4t" event={"ID":"bcc0d689-48b6-467e-bdd0-6a03878a52b6","Type":"ContainerStarted","Data":"571da96cb8b6a018b0047426b1c5561018ac9f326cf4ad56f325132b4fcbec4b"} Nov 04 03:49:06.887982 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.887887 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5xh74" event={"ID":"21ac7748-5e3d-427e-bd15-93cf4754cb0e","Type":"ContainerStarted","Data":"8ddf93747976c4469160b51e7421723397597f92296ee22a6001853599bd527b"} Nov 04 03:49:06.894210 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.894127 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vjhfk" event={"ID":"3fbef9e3-d5de-4a35-8745-2b1dc357c52b","Type":"ContainerStarted","Data":"75dd49f74d6e64dd95776fe7d6053b7f8010387a3ac35b7c7413713dbc41ba5e"} Nov 04 03:49:06.895809 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.895732 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" event={"ID":"d2052e7f-5ee8-4066-bb29-cd227fa0965b","Type":"ContainerStarted","Data":"cd27c2ebd27cf211c4d38ad9efc9a443a46483c9bc3aec8bdc605979b7ab4467"} Nov 04 03:49:06.899696 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.898801 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-8v2sd" podStartSLOduration=386.898785721 podStartE2EDuration="6m26.898785721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:06.897637365 +0000 UTC m=+31.737547873" watchObservedRunningTime="2026-11-04 03:49:06.898785721 +0000 UTC m=+31.738696231" Nov 04 03:49:06.899696 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.899331 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-qq74s" podStartSLOduration=386.899321553 podStartE2EDuration="6m26.899321553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:06.878867979 +0000 UTC m=+31.718778490" watchObservedRunningTime="2026-11-04 03:49:06.899321553 +0000 UTC m=+31.739232061" Nov 04 03:49:06.906772 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.906748 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wc5zp" event={"ID":"1c2ac1f5-6e08-48e3-ac83-a55f9181f4be","Type":"ContainerStarted","Data":"553b2562f47a0bad80e7d66848605dc8ea27c9b42204f27dcb43f23f4eb32849"} Nov 04 03:49:06.925969 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:06.925905 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-wc5zp" podStartSLOduration=386.925891943 podStartE2EDuration="6m26.925891943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:06.925217791 +0000 UTC m=+31.765128299" watchObservedRunningTime="2026-11-04 03:49:06.925891943 +0000 UTC m=+31.765802451" Nov 04 03:49:08.803381 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:08.802767 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:08.803381 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:08.802882 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f37b227d-fc9c-4a81-85b1-62c04243ad83-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2hhp2\" (UID: \"f37b227d-fc9c-4a81-85b1-62c04243ad83\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2hhp2" Nov 04 03:49:08.803381 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:08.802968 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-metrics-certs\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:08.803381 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:08.802996 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0df87f5d-31f4-4571-82b6-fe75c434ff27-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qmh6r\" (UID: \"0df87f5d-31f4-4571-82b6-fe75c434ff27\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:08.803381 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:08.803142 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 03:49:08.803381 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:08.803196 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0df87f5d-31f4-4571-82b6-fe75c434ff27-cluster-monitoring-operator-tls podName:0df87f5d-31f4-4571-82b6-fe75c434ff27 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:12.803179211 +0000 UTC m=+37.643089728 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0df87f5d-31f4-4571-82b6-fe75c434ff27-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qmh6r" (UID: "0df87f5d-31f4-4571-82b6-fe75c434ff27") : secret "cluster-monitoring-operator-tls" not found Nov 04 03:49:08.803381 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:08.803252 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 03:49:08.803381 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:08.803283 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f37b227d-fc9c-4a81-85b1-62c04243ad83-networking-console-plugin-cert podName:f37b227d-fc9c-4a81-85b1-62c04243ad83 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:12.803273531 +0000 UTC m=+37.643184020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f37b227d-fc9c-4a81-85b1-62c04243ad83-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2hhp2" (UID: "f37b227d-fc9c-4a81-85b1-62c04243ad83") : secret "networking-console-plugin-cert" not found Nov 04 03:49:08.803381 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:08.803321 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle podName:1b81b9bc-4eb5-48a7-98e1-6c7a98117f57 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:12.803296614 +0000 UTC m=+37.643207104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle") pod "router-default-78ccf8f879-s4sx4" (UID: "1b81b9bc-4eb5-48a7-98e1-6c7a98117f57") : configmap references non-existent config key: service-ca.crt Nov 04 03:49:08.803381 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:08.803330 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 03:49:08.803381 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:08.803361 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-metrics-certs podName:1b81b9bc-4eb5-48a7-98e1-6c7a98117f57 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:12.803354379 +0000 UTC m=+37.643264922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-metrics-certs") pod "router-default-78ccf8f879-s4sx4" (UID: "1b81b9bc-4eb5-48a7-98e1-6c7a98117f57") : secret "router-metrics-certs-default" not found Nov 04 03:49:08.903822 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:08.903776 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:08.904036 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:08.903888 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:08.904036 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:08.903973 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:08.904036 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:08.904023 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:08.904195 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:08.904180 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 03:49:08.904291 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:08.904243 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls podName:30f18911-ab5a-44e3-88b2-6508325ceb39 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:12.904224432 +0000 UTC m=+37.744134934 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6x2fz" (UID: "30f18911-ab5a-44e3-88b2-6508325ceb39") : secret "samples-operator-tls" not found Nov 04 03:49:08.904667 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:08.904646 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 03:49:08.904753 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:08.904699 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls podName:7b632ca9-a695-450e-a1a9-b5357ebf4d7d nodeName:}" failed. No retries permitted until 2026-11-04 03:49:12.904685899 +0000 UTC m=+37.744596399 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls") pod "dns-default-pvrd4" (UID: "7b632ca9-a695-450e-a1a9-b5357ebf4d7d") : secret "dns-default-metrics-tls" not found Nov 04 03:49:08.904813 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:08.904763 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:49:08.904813 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:08.904773 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8648b894c6-t5rf2: secret "image-registry-tls" not found Nov 04 03:49:08.904813 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:08.904806 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls podName:3ce83f95-758c-4fd2-9678-441fe4309da7 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:12.90479422 +0000 UTC m=+37.744704709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls") pod "image-registry-8648b894c6-t5rf2" (UID: "3ce83f95-758c-4fd2-9678-441fe4309da7") : secret "image-registry-tls" not found Nov 04 03:49:08.904987 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:08.904861 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:49:08.904987 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:08.904871 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6448ff747c-dlpnw: secret "image-registry-tls" not found Nov 04 03:49:08.904987 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:08.904900 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls podName:363d8742-c0a7-405c-9683-7b12afbd8bcc nodeName:}" failed. No retries permitted until 2026-11-04 03:49:12.904889867 +0000 UTC m=+37.744800367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls") pod "image-registry-6448ff747c-dlpnw" (UID: "363d8742-c0a7-405c-9683-7b12afbd8bcc") : secret "image-registry-tls" not found Nov 04 03:49:08.923963 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:08.923492 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vjhfk" event={"ID":"3fbef9e3-d5de-4a35-8745-2b1dc357c52b","Type":"ContainerStarted","Data":"2a780c4ab4f8e86adac768acb1647e57ec25d1165418e561c0cec6864bc0a469"} Nov 04 03:49:08.943146 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:08.943090 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vjhfk" podStartSLOduration=353.901141347 podStartE2EDuration="5m55.943072195s" totalImagesPullingTime="2.041930848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:43:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:08.941872215 +0000 UTC m=+33.781782734" watchObservedRunningTime="2026-11-04 03:49:08.943072195 +0000 UTC m=+33.782982703" Nov 04 03:49:09.411625 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:09.410548 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:49:09.411625 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:09.410626 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2fj\" (UniqueName: \"kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj\") pod \"network-check-target-pbsd4\" (UID: \"391e5382-7119-4085-9939-cd81fc8b2d73\") " pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:49:09.411625 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:09.410650 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs\") pod \"network-metrics-daemon-mwzdx\" (UID: \"9ab83f7b-3396-4cc7-875e-c2cc62696ff5\") " pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:49:09.411625 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:09.410797 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 03:49:09.411625 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:09.410859 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs podName:9ab83f7b-3396-4cc7-875e-c2cc62696ff5 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:41.410840793 +0000 UTC m=+66.250751292 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs") pod "network-metrics-daemon-mwzdx" (UID: "9ab83f7b-3396-4cc7-875e-c2cc62696ff5") : secret "metrics-daemon-secret" not found Nov 04 03:49:09.415466 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:09.415442 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:49:09.422953 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:09.422885 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq2fj\" (UniqueName: \"kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj\") pod \"network-check-target-pbsd4\" (UID: \"391e5382-7119-4085-9939-cd81fc8b2d73\") " pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:49:09.619637 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:09.619439 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:49:09.676809 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:09.676774 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:49:09.927303 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:09.927157 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-59fsx"] Nov 04 03:49:10.009435 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:10.009412 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pbsd4"] Nov 04 03:49:10.136388 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:49:10.136354 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26bb99fd_d223_4642_b147_b22bebd00835.slice/crio-09df2a931fa2c83c00586babb549bfbbe214e7590b4938a62dbfbc2838349918 WatchSource:0}: Error finding container 09df2a931fa2c83c00586babb549bfbbe214e7590b4938a62dbfbc2838349918: Status 404 returned error can't find the container with id 09df2a931fa2c83c00586babb549bfbbe214e7590b4938a62dbfbc2838349918 Nov 04 03:49:10.185759 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:49:10.185687 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391e5382_7119_4085_9939_cd81fc8b2d73.slice/crio-bd4f620630a7a6a3d7560eff980ce5bd4108983f766deef2342e0541d416e3b1 WatchSource:0}: Error finding container bd4f620630a7a6a3d7560eff980ce5bd4108983f766deef2342e0541d416e3b1: Status 404 returned error can't find the container with id bd4f620630a7a6a3d7560eff980ce5bd4108983f766deef2342e0541d416e3b1 Nov 04 03:49:10.933891 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:10.933845 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" event={"ID":"3d313ca1-e640-42d9-a9f7-5e3e8918dc77","Type":"ContainerStarted","Data":"26eda4d6c4411a06bb40bbe3f7aba060db366a4d767a3a7310eeb374b7d4a159"} Nov 04 03:49:10.935847 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:10.935818 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" event={"ID":"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6","Type":"ContainerStarted","Data":"f1420a8ed802f516dfb3ee056bf0c85b250d5d157c9784d7a58da3d0fc26f650"} Nov 04 03:49:10.935988 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:10.935961 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:10.937526 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:10.937497 2584 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-rr5bv container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body= Nov 04 03:49:10.937621 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:10.937542 2584 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" podUID="9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" Nov 04 03:49:10.937746 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:10.937722 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" event={"ID":"d2052e7f-5ee8-4066-bb29-cd227fa0965b","Type":"ContainerStarted","Data":"88e5477e3067072f083da4dc4eacbb4b27e67337a6fd9221c9984b5e666c4f43"} Nov 04 03:49:10.939538 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:10.939482 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pbsd4" event={"ID":"391e5382-7119-4085-9939-cd81fc8b2d73","Type":"ContainerStarted","Data":"bd4f620630a7a6a3d7560eff980ce5bd4108983f766deef2342e0541d416e3b1"} Nov 04 03:49:10.941363 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:10.941339 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-gm5s7" event={"ID":"3b456f4f-071a-4e5f-b8b6-51c3ee790656","Type":"ContainerStarted","Data":"fd75078afffb49d9c74a8663412e6c36588f2b16945533cde860aa1d8f4d637a"} Nov 04 03:49:10.942628 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:10.942607 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-59fsx" event={"ID":"26bb99fd-d223-4642-b147-b22bebd00835","Type":"ContainerStarted","Data":"09df2a931fa2c83c00586babb549bfbbe214e7590b4938a62dbfbc2838349918"} Nov 04 03:49:10.953414 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:10.953369 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" podStartSLOduration=353.119251623 podStartE2EDuration="5m57.953353369s" totalImagesPullingTime="4.834101746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:43:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:10.951797265 +0000 UTC m=+35.791707831" watchObservedRunningTime="2026-11-04 03:49:10.953353369 +0000 UTC m=+35.793263875" Nov 04 03:49:10.976369 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:10.976332 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" podStartSLOduration=353.202873142 podStartE2EDuration="5m57.976320232s" totalImagesPullingTime="4.77344709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:43:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:10.969326769 +0000 UTC m=+35.809237279" watchObservedRunningTime="2026-11-04 03:49:10.976320232 +0000 UTC m=+35.816230738" Nov 04 03:49:10.994842 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:10.994678 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" podStartSLOduration=353.232234863 podStartE2EDuration="5m57.994665796s" totalImagesPullingTime="4.762430933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:43:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:10.993759629 +0000 UTC m=+35.833670136" watchObservedRunningTime="2026-11-04 03:49:10.994665796 +0000 UTC m=+35.834576302" Nov 04 03:49:11.014981 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.014910 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-gm5s7" podStartSLOduration=353.189300104 podStartE2EDuration="5m58.014896445s" totalImagesPullingTime="4.825596341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:43:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:11.014336113 +0000 UTC m=+35.854246620" watchObservedRunningTime="2026-11-04 03:49:11.014896445 +0000 UTC m=+35.854806945" Nov 04 03:49:11.233811 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.232305 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zfjhp"] Nov 04 03:49:11.268343 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.259116 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-xwdp2-kwf4s"] Nov 04 03:49:11.268343 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.260407 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:11.275688 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.275591 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n475p\"" Nov 04 03:49:11.275688 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.275635 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 03:49:11.275688 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.275664 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 03:49:11.288851 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.288417 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zfjhp"] Nov 04 03:49:11.288851 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.288466 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xwdp2-kwf4s"] Nov 04 03:49:11.288851 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.288552 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:11.433676 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.433643 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsvdh\" (UniqueName: \"kubernetes.io/projected/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-kube-api-access-lsvdh\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:11.433836 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.433725 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78e55dc9-354d-455f-9d02-b3cfd5885d65-data-volume\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:11.433836 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.433758 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w8m8\" (UniqueName: \"kubernetes.io/projected/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-api-access-6w8m8\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:11.433836 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.433788 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:11.434012 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.433880 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78e55dc9-354d-455f-9d02-b3cfd5885d65-crio-socket\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:11.434012 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.433914 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-trusted-ca-bundle\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:11.434012 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.433958 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-archives-path\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:11.434148 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.434054 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-service-ca-bundle\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:11.434148 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.434123 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:11.535056 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.535020 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:11.535159 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.535086 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsvdh\" (UniqueName: \"kubernetes.io/projected/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-kube-api-access-lsvdh\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:11.535159 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.535132 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78e55dc9-354d-455f-9d02-b3cfd5885d65-data-volume\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:11.535159 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.535155 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w8m8\" (UniqueName: \"kubernetes.io/projected/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-api-access-6w8m8\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:11.535252 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.535172 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:11.535252 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.535223 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78e55dc9-354d-455f-9d02-b3cfd5885d65-crio-socket\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:11.535252 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.535241 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-trusted-ca-bundle\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:11.535338 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.535260 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-archives-path\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:11.535338 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.535313 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-service-ca-bundle\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:11.535642 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:11.535616 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 03:49:11.535769 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:11.535683 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-rbac-proxy-cm podName:78e55dc9-354d-455f-9d02-b3cfd5885d65 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:12.035664929 +0000 UTC m=+36.875575429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zfjhp" (UID: "78e55dc9-354d-455f-9d02-b3cfd5885d65") : configmap "kube-rbac-proxy" not found Nov 04 03:49:11.535867 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.535847 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-service-ca-bundle\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:11.535936 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.535875 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78e55dc9-354d-455f-9d02-b3cfd5885d65-data-volume\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:11.535936 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.535892 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78e55dc9-354d-455f-9d02-b3cfd5885d65-crio-socket\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:11.536033 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:11.535989 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 03:49:11.536085 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:11.536036 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls podName:78e55dc9-354d-455f-9d02-b3cfd5885d65 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:12.036022077 +0000 UTC m=+36.875932569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zfjhp" (UID: "78e55dc9-354d-455f-9d02-b3cfd5885d65") : secret "insights-runtime-extractor-tls" not found Nov 04 03:49:11.536139 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.536118 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-archives-path\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:11.536442 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.536425 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-trusted-ca-bundle\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:11.553279 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.553252 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsvdh\" (UniqueName: \"kubernetes.io/projected/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-kube-api-access-lsvdh\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:11.553474 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.553452 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w8m8\" (UniqueName: \"kubernetes.io/projected/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-api-access-6w8m8\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:11.663958 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.659505 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:11.949956 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.949870 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/0.log" Nov 04 03:49:11.949956 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.949911 2584 generic.go:356] "Generic (PLEG): container finished" podID="9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6" containerID="f1420a8ed802f516dfb3ee056bf0c85b250d5d157c9784d7a58da3d0fc26f650" exitCode=255 Nov 04 03:49:11.950417 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.950031 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" event={"ID":"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6","Type":"ContainerDied","Data":"f1420a8ed802f516dfb3ee056bf0c85b250d5d157c9784d7a58da3d0fc26f650"} Nov 04 03:49:11.950417 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.950320 2584 scope.go:122] "RemoveContainer" containerID="f1420a8ed802f516dfb3ee056bf0c85b250d5d157c9784d7a58da3d0fc26f650" Nov 04 03:49:11.952669 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.952649 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-chk4t" event={"ID":"bcc0d689-48b6-467e-bdd0-6a03878a52b6","Type":"ContainerStarted","Data":"cd781305a6aa90099bb60a7d3da072e82c785c056b5d774c9df6539bd14ff552"} Nov 04 03:49:11.955545 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.955521 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5xh74" event={"ID":"21ac7748-5e3d-427e-bd15-93cf4754cb0e","Type":"ContainerStarted","Data":"2a599f4d169bf8d75c69d318f7907d0ade5463679ce6faaef1e8da9c23c8b8f1"} Nov 04 03:49:11.957522 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.957490 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pbsd4" event={"ID":"391e5382-7119-4085-9939-cd81fc8b2d73","Type":"ContainerStarted","Data":"f79e733393080272f5dc5148f9dff452b9ab8b63eeb7167654cd6a8b7561a89b"} Nov 04 03:49:11.958670 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.958654 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:49:11.978664 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:11.978643 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xwdp2-kwf4s"] Nov 04 03:49:11.980979 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:49:11.980705 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341692ff_c7ac_4387_b9a3_d41ddd8f2ad3.slice/crio-747fe1cc8b3517919d67735d1b24ebf8bc097f87f185660ed9f0c0bd64c45773 WatchSource:0}: Error finding container 747fe1cc8b3517919d67735d1b24ebf8bc097f87f185660ed9f0c0bd64c45773: Status 404 returned error can't find the container with id 747fe1cc8b3517919d67735d1b24ebf8bc097f87f185660ed9f0c0bd64c45773 Nov 04 03:49:12.029267 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.029213 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-chk4t" podStartSLOduration=1.926260283 podStartE2EDuration="7.02919421s" totalImagesPullingTime="5.102933927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:12.028116476 +0000 UTC m=+36.868026984" watchObservedRunningTime="2026-11-04 03:49:12.02919421 +0000 UTC m=+36.869104719" Nov 04 03:49:12.041887 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.041867 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:12.042000 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.041982 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:12.042122 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.042106 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 03:49:12.042184 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.042176 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls podName:78e55dc9-354d-455f-9d02-b3cfd5885d65 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:13.042149038 +0000 UTC m=+37.882059524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zfjhp" (UID: "78e55dc9-354d-455f-9d02-b3cfd5885d65") : secret "insights-runtime-extractor-tls" not found Nov 04 03:49:12.042396 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.042335 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 03:49:12.042481 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.042400 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-rbac-proxy-cm podName:78e55dc9-354d-455f-9d02-b3cfd5885d65 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:13.042385985 +0000 UTC m=+37.882296470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zfjhp" (UID: "78e55dc9-354d-455f-9d02-b3cfd5885d65") : configmap "kube-rbac-proxy" not found Nov 04 03:49:12.072998 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.066753 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pbsd4" podStartSLOduration=36.106589212 podStartE2EDuration="37.066734608s" totalImagesPullingTime="960.145396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:12.063214088 +0000 UTC m=+36.903124599" watchObservedRunningTime="2026-11-04 03:49:12.066734608 +0000 UTC m=+36.906645118" Nov 04 03:49:12.240343 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.240192 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5xh74" podStartSLOduration=351.652680667 podStartE2EDuration="5m57.240170386s" totalImagesPullingTime="5.587489719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:43:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:12.097703784 +0000 UTC m=+36.937614311" watchObservedRunningTime="2026-11-04 03:49:12.240170386 +0000 UTC m=+37.080080894" Nov 04 03:49:12.241229 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.241194 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-tr6w8"] Nov 04 03:49:12.266027 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.265988 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-tr6w8"] Nov 04 03:49:12.266171 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.266127 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" Nov 04 03:49:12.276721 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.276697 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 04 03:49:12.276892 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.276754 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 04 03:49:12.277027 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.276818 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 04 03:49:12.289105 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.289081 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 04 03:49:12.289204 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.289126 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pb77x\"" Nov 04 03:49:12.450731 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.450693 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/366aeead-59de-4180-8d14-e8b821d04e22-signing-cabundle\") pod \"service-ca-5fffc97bdf-tr6w8\" (UID: \"366aeead-59de-4180-8d14-e8b821d04e22\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" Nov 04 03:49:12.450902 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.450839 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8852w\" (UniqueName: \"kubernetes.io/projected/366aeead-59de-4180-8d14-e8b821d04e22-kube-api-access-8852w\") pod \"service-ca-5fffc97bdf-tr6w8\" (UID: \"366aeead-59de-4180-8d14-e8b821d04e22\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" Nov 04 03:49:12.450985 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.450907 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/366aeead-59de-4180-8d14-e8b821d04e22-signing-key\") pod \"service-ca-5fffc97bdf-tr6w8\" (UID: \"366aeead-59de-4180-8d14-e8b821d04e22\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" Nov 04 03:49:12.551792 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.551703 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/366aeead-59de-4180-8d14-e8b821d04e22-signing-cabundle\") pod \"service-ca-5fffc97bdf-tr6w8\" (UID: \"366aeead-59de-4180-8d14-e8b821d04e22\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" Nov 04 03:49:12.551972 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.551832 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8852w\" (UniqueName: \"kubernetes.io/projected/366aeead-59de-4180-8d14-e8b821d04e22-kube-api-access-8852w\") pod \"service-ca-5fffc97bdf-tr6w8\" (UID: \"366aeead-59de-4180-8d14-e8b821d04e22\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" Nov 04 03:49:12.551972 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.551876 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/366aeead-59de-4180-8d14-e8b821d04e22-signing-key\") pod \"service-ca-5fffc97bdf-tr6w8\" (UID: \"366aeead-59de-4180-8d14-e8b821d04e22\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" Nov 04 03:49:12.552400 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.552375 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/366aeead-59de-4180-8d14-e8b821d04e22-signing-cabundle\") pod \"service-ca-5fffc97bdf-tr6w8\" (UID: \"366aeead-59de-4180-8d14-e8b821d04e22\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" Nov 04 03:49:12.554945 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.554815 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/366aeead-59de-4180-8d14-e8b821d04e22-signing-key\") pod \"service-ca-5fffc97bdf-tr6w8\" (UID: \"366aeead-59de-4180-8d14-e8b821d04e22\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" Nov 04 03:49:12.564537 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.564514 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8852w\" (UniqueName: \"kubernetes.io/projected/366aeead-59de-4180-8d14-e8b821d04e22-kube-api-access-8852w\") pod \"service-ca-5fffc97bdf-tr6w8\" (UID: \"366aeead-59de-4180-8d14-e8b821d04e22\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" Nov 04 03:49:12.598429 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.598401 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" Nov 04 03:49:12.855069 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.855033 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:12.855226 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.855110 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f37b227d-fc9c-4a81-85b1-62c04243ad83-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2hhp2\" (UID: \"f37b227d-fc9c-4a81-85b1-62c04243ad83\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2hhp2" Nov 04 03:49:12.855226 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.855201 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-metrics-certs\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:12.855345 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.855238 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle podName:1b81b9bc-4eb5-48a7-98e1-6c7a98117f57 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:20.855215146 +0000 UTC m=+45.695125638 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle") pod "router-default-78ccf8f879-s4sx4" (UID: "1b81b9bc-4eb5-48a7-98e1-6c7a98117f57") : configmap references non-existent config key: service-ca.crt Nov 04 03:49:12.855345 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.855285 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 03:49:12.855345 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.855295 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0df87f5d-31f4-4571-82b6-fe75c434ff27-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qmh6r\" (UID: \"0df87f5d-31f4-4571-82b6-fe75c434ff27\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:12.855345 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.855335 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-metrics-certs podName:1b81b9bc-4eb5-48a7-98e1-6c7a98117f57 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:20.855318481 +0000 UTC m=+45.695228966 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-metrics-certs") pod "router-default-78ccf8f879-s4sx4" (UID: "1b81b9bc-4eb5-48a7-98e1-6c7a98117f57") : secret "router-metrics-certs-default" not found Nov 04 03:49:12.855536 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.855373 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 03:49:12.855536 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.855403 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0df87f5d-31f4-4571-82b6-fe75c434ff27-cluster-monitoring-operator-tls podName:0df87f5d-31f4-4571-82b6-fe75c434ff27 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:20.855395674 +0000 UTC m=+45.695306159 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0df87f5d-31f4-4571-82b6-fe75c434ff27-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qmh6r" (UID: "0df87f5d-31f4-4571-82b6-fe75c434ff27") : secret "cluster-monitoring-operator-tls" not found Nov 04 03:49:12.855536 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.855287 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 03:49:12.855536 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.855439 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f37b227d-fc9c-4a81-85b1-62c04243ad83-networking-console-plugin-cert podName:f37b227d-fc9c-4a81-85b1-62c04243ad83 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:20.855429047 +0000 UTC m=+45.695339533 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f37b227d-fc9c-4a81-85b1-62c04243ad83-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2hhp2" (UID: "f37b227d-fc9c-4a81-85b1-62c04243ad83") : secret "networking-console-plugin-cert" not found Nov 04 03:49:12.894359 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.894310 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-tr6w8"] Nov 04 03:49:12.901086 ip-10-0-138-9 kubenswrapper[2584]: W1104 03:49:12.901058 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366aeead_59de_4180_8d14_e8b821d04e22.slice/crio-43d5131a19d1a624b54ac101f804cda981ed9633089b31b169023d5bb60395b7 WatchSource:0}: Error finding container 43d5131a19d1a624b54ac101f804cda981ed9633089b31b169023d5bb60395b7: Status 404 returned error can't find the container with id 43d5131a19d1a624b54ac101f804cda981ed9633089b31b169023d5bb60395b7 Nov 04 03:49:12.955784 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.955759 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:12.956118 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.955896 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:12.956118 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.955901 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 03:49:12.956118 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.955977 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls podName:7b632ca9-a695-450e-a1a9-b5357ebf4d7d nodeName:}" failed. No retries permitted until 2026-11-04 03:49:20.955958728 +0000 UTC m=+45.795869213 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls") pod "dns-default-pvrd4" (UID: "7b632ca9-a695-450e-a1a9-b5357ebf4d7d") : secret "dns-default-metrics-tls" not found Nov 04 03:49:12.956118 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.956014 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:12.956118 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.956034 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:49:12.956118 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.956048 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8648b894c6-t5rf2: secret "image-registry-tls" not found Nov 04 03:49:12.956118 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.956050 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:12.956118 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.956091 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls podName:3ce83f95-758c-4fd2-9678-441fe4309da7 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:20.956076934 +0000 UTC m=+45.795987434 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls") pod "image-registry-8648b894c6-t5rf2" (UID: "3ce83f95-758c-4fd2-9678-441fe4309da7") : secret "image-registry-tls" not found Nov 04 03:49:12.956118 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.956115 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 03:49:12.956421 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.956145 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls podName:30f18911-ab5a-44e3-88b2-6508325ceb39 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:20.956136469 +0000 UTC m=+45.796046968 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6x2fz" (UID: "30f18911-ab5a-44e3-88b2-6508325ceb39") : secret "samples-operator-tls" not found Nov 04 03:49:12.956421 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.956218 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:49:12.956421 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.956232 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6448ff747c-dlpnw: secret "image-registry-tls" not found Nov 04 03:49:12.956421 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.956276 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls podName:363d8742-c0a7-405c-9683-7b12afbd8bcc nodeName:}" failed. No retries permitted until 2026-11-04 03:49:20.956260508 +0000 UTC m=+45.796170993 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls") pod "image-registry-6448ff747c-dlpnw" (UID: "363d8742-c0a7-405c-9683-7b12afbd8bcc") : secret "image-registry-tls" not found Nov 04 03:49:12.961945 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.961902 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" event={"ID":"366aeead-59de-4180-8d14-e8b821d04e22","Type":"ContainerStarted","Data":"43d5131a19d1a624b54ac101f804cda981ed9633089b31b169023d5bb60395b7"} Nov 04 03:49:12.963504 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.963480 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" event={"ID":"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3","Type":"ContainerStarted","Data":"0d406fb7c7275db00231e338cea8ff883fb5db9637204678bfffa553cf628eee"} Nov 04 03:49:12.963602 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.963527 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" event={"ID":"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3","Type":"ContainerStarted","Data":"747fe1cc8b3517919d67735d1b24ebf8bc097f87f185660ed9f0c0bd64c45773"} Nov 04 03:49:12.965307 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.965289 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/1.log" Nov 04 03:49:12.965780 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.965763 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/0.log" Nov 04 03:49:12.965866 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.965795 2584 generic.go:356] "Generic (PLEG): container finished" podID="9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6" containerID="a269cc6bda79ea6e7251d9cfa350a30df870c64031267278c93cb9a0f5bd17e5" exitCode=255 Nov 04 03:49:12.965940 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.965860 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" event={"ID":"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6","Type":"ContainerDied","Data":"a269cc6bda79ea6e7251d9cfa350a30df870c64031267278c93cb9a0f5bd17e5"} Nov 04 03:49:12.965940 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.965931 2584 scope.go:122] "RemoveContainer" containerID="f1420a8ed802f516dfb3ee056bf0c85b250d5d157c9784d7a58da3d0fc26f650" Nov 04 03:49:12.966136 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:12.966121 2584 scope.go:122] "RemoveContainer" containerID="a269cc6bda79ea6e7251d9cfa350a30df870c64031267278c93cb9a0f5bd17e5" Nov 04 03:49:12.966334 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:12.966313 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rr5bv_openshift-console-operator(9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" podUID="9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6" Nov 04 03:49:13.009454 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:13.009401 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" podStartSLOduration=2.009385041 podStartE2EDuration="2.009385041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:13.008820101 +0000 UTC m=+37.848730613" watchObservedRunningTime="2026-11-04 03:49:13.009385041 +0000 UTC m=+37.849295548" Nov 04 03:49:13.057647 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:13.057610 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:13.057855 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:13.057834 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:13.058063 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:13.058043 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 03:49:13.058142 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:13.058110 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls podName:78e55dc9-354d-455f-9d02-b3cfd5885d65 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:15.058091245 +0000 UTC m=+39.898001750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zfjhp" (UID: "78e55dc9-354d-455f-9d02-b3cfd5885d65") : secret "insights-runtime-extractor-tls" not found Nov 04 03:49:13.058216 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:13.058142 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 03:49:13.058216 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:13.058187 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-rbac-proxy-cm podName:78e55dc9-354d-455f-9d02-b3cfd5885d65 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:15.058173847 +0000 UTC m=+39.898084346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zfjhp" (UID: "78e55dc9-354d-455f-9d02-b3cfd5885d65") : configmap "kube-rbac-proxy" not found Nov 04 03:49:13.610400 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:13.610375 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c8xnp_c12382e2-90c4-45bf-bdd8-27653073cc5d/dns-node-resolver/0.log" Nov 04 03:49:13.977020 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:13.976313 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" event={"ID":"366aeead-59de-4180-8d14-e8b821d04e22","Type":"ContainerStarted","Data":"87e73a0fa7142c660f8db428fd5320d9118077a2787810af9933c811a1d4a360"} Nov 04 03:49:13.981669 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:13.980629 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/1.log" Nov 04 03:49:13.981669 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:13.981245 2584 scope.go:122] "RemoveContainer" containerID="a269cc6bda79ea6e7251d9cfa350a30df870c64031267278c93cb9a0f5bd17e5" Nov 04 03:49:13.981669 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:13.981399 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rr5bv_openshift-console-operator(9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" podUID="9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6" Nov 04 03:49:14.003227 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:14.002277 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" podStartSLOduration=2.002257714 podStartE2EDuration="2.002257714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:14.00112711 +0000 UTC m=+38.841037623" watchObservedRunningTime="2026-11-04 03:49:14.002257714 +0000 UTC m=+38.842168223" Nov 04 03:49:14.810964 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:14.810908 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jkpjj_795d3139-8d19-4852-bd9c-3c60b01a97f6/node-ca/0.log" Nov 04 03:49:15.082665 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:15.082582 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:15.083137 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:15.082748 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 03:49:15.083137 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:15.082812 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls podName:78e55dc9-354d-455f-9d02-b3cfd5885d65 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:19.082796996 +0000 UTC m=+43.922707485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zfjhp" (UID: "78e55dc9-354d-455f-9d02-b3cfd5885d65") : secret "insights-runtime-extractor-tls" not found Nov 04 03:49:15.083137 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:15.082837 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:15.083137 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:15.083067 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 03:49:15.083137 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:15.083118 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-rbac-proxy-cm podName:78e55dc9-354d-455f-9d02-b3cfd5885d65 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:19.083101938 +0000 UTC m=+43.923012438 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zfjhp" (UID: "78e55dc9-354d-455f-9d02-b3cfd5885d65") : configmap "kube-rbac-proxy" not found Nov 04 03:49:15.210181 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:15.210135 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-chk4t_bcc0d689-48b6-467e-bdd0-6a03878a52b6/serve-healthcheck-canary/0.log" Nov 04 03:49:15.459190 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:15.459107 2584 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:15.459557 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:15.459539 2584 scope.go:122] "RemoveContainer" containerID="a269cc6bda79ea6e7251d9cfa350a30df870c64031267278c93cb9a0f5bd17e5" Nov 04 03:49:15.459773 ip-10-0-138-9 kubenswrapper[2584]: E1104 03:49:15.459752 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rr5bv_openshift-console-operator(9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" podUID="9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6" Nov 04 03:49:15.989182 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:15.989144 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-59fsx" event={"ID":"26bb99fd-d223-4642-b147-b22bebd00835","Type":"ContainerStarted","Data":"60fbc32e7baf29d1bd1c19dedc1468fc0264b5a5e48c3959ff8d392c09d8ca67"} Nov 04 03:49:16.007331 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:16.007278 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-59fsx" podStartSLOduration=34.304646906 podStartE2EDuration="40.007266035s" totalImagesPullingTime="5.702619129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:16.005847484 +0000 UTC m=+40.845757991" watchObservedRunningTime="2026-11-04 03:49:16.007266035 +0000 UTC m=+40.847176542" Nov 04 03:49:16.149776 ip-10-0-138-9 kubenswrapper[2584]: I1104 03:49:16.149598 2584 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 03:49:16.149796 ip-10-0-138-9 systemd[1]: Stopping Kubernetes Kubelet... Nov 04 03:49:16.160321 ip-10-0-138-9 systemd[1]: kubelet.service: Deactivated successfully. Nov 04 03:49:16.160486 ip-10-0-138-9 systemd[1]: Stopped Kubernetes Kubelet. Nov 04 03:49:16.160556 ip-10-0-138-9 systemd[1]: kubelet.service: Consumed 3.048s CPU time. Nov 04 03:49:16.168564 ip-10-0-138-9 systemd[1]: Starting Kubernetes Kubelet... Nov 04 03:49:16.239399 ip-10-0-138-9 kubenswrapper[4966]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 03:49:16.239399 ip-10-0-138-9 kubenswrapper[4966]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 03:49:16.239399 ip-10-0-138-9 kubenswrapper[4966]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 03:49:16.239399 ip-10-0-138-9 kubenswrapper[4966]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 03:49:16.242781 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242662 4966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:49:16.242781 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242781 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:49:16.242864 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242786 4966 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:49:16.242864 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242791 4966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:49:16.242864 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242796 4966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:49:16.242864 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242801 4966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:49:16.242864 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242805 4966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:49:16.242864 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242809 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:49:16.242864 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242813 4966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:49:16.242864 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242817 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:49:16.242864 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242821 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:49:16.242864 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242825 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:49:16.242864 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242829 4966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:49:16.242864 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242833 4966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:49:16.242864 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242837 4966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:49:16.242864 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242842 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:49:16.242864 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242846 4966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:49:16.242864 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242850 4966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:49:16.242864 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242863 4966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:49:16.242864 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242868 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:49:16.243398 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242873 4966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:49:16.243398 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242877 4966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:49:16.243398 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242884 4966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:49:16.243398 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242890 4966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:49:16.243398 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242894 4966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:49:16.243398 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242899 4966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:49:16.243398 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242903 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:49:16.243398 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242907 4966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:49:16.243398 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242911 4966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:49:16.243398 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242930 4966 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:49:16.243398 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242935 4966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:49:16.243398 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242939 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:49:16.243398 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242943 4966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:49:16.243398 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242947 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:49:16.243398 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242951 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:49:16.243398 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242955 4966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:49:16.243398 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242959 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:49:16.243398 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242963 4966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:49:16.243398 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242967 4966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242971 4966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242975 4966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242979 4966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242983 4966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242988 4966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242992 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.242997 4966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243001 4966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243005 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243011 4966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243015 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243019 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243023 4966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243028 4966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243032 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243036 4966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243040 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243044 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243048 4966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:49:16.243878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243053 4966 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243057 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243061 4966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243065 4966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243069 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243073 4966 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243077 4966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243081 4966 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243085 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243089 4966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243093 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243097 4966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243101 4966 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243106 4966 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243110 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243114 4966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243118 4966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243132 4966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243136 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243142 4966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:49:16.244419 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243146 4966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243150 4966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243154 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243159 4966 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243163 4966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243167 4966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243171 4966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243175 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243179 4966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243184 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243187 4966 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243190 4966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243193 4966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243196 4966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243199 4966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243202 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243205 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243208 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243210 4966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243213 4966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:49:16.244911 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243216 4966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:49:16.245415 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243219 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:49:16.245415 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243221 4966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:49:16.245415 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243224 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:49:16.245415 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243227 4966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:49:16.245415 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243232 4966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:49:16.245415 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243236 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:49:16.245415 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.243240 4966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:49:16.245415 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244027 4966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:49:16.245415 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244035 4966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:49:16.245415 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244039 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:49:16.245415 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244042 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:49:16.245415 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244045 4966 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:49:16.245415 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244048 4966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:49:16.245415 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244051 4966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:49:16.245415 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244054 4966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:49:16.245415 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244057 4966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:49:16.245415 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244061 4966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:49:16.245415 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244064 4966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:49:16.245415 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244067 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244070 4966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244072 4966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244075 4966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244077 4966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244080 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244083 4966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244086 4966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244089 4966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244092 4966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244094 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244097 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244100 4966 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244103 4966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244105 4966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244108 4966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244111 4966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244114 4966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244117 4966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244120 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:49:16.245899 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244122 4966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244125 4966 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244128 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244130 4966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244133 4966 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244136 4966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244138 4966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244141 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244143 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244146 4966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244153 4966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244158 4966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244160 4966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244163 4966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244166 4966 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244169 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244171 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244174 4966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244177 4966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244179 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:49:16.246426 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244182 4966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:49:16.246988 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244185 4966 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:49:16.246988 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244187 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:49:16.246988 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244190 4966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:49:16.246988 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244193 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:49:16.246988 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244196 4966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:49:16.246988 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244199 4966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:49:16.246988 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244201 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:49:16.246988 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244204 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:49:16.246988 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244207 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:49:16.246988 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244210 4966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:49:16.246988 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244212 4966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:49:16.246988 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244215 4966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:49:16.246988 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244218 4966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:49:16.246988 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244220 4966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:49:16.246988 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244223 4966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:49:16.246988 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244225 4966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:49:16.246988 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244228 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:49:16.246988 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244231 4966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:49:16.246988 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244233 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:49:16.247457 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244238 4966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:49:16.247457 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244242 4966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:49:16.247457 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244246 4966 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:49:16.247457 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244250 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:49:16.247457 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244253 4966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:49:16.247457 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244256 4966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:49:16.247457 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244260 4966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:49:16.247457 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244262 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:49:16.247457 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244265 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:49:16.247457 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244268 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:49:16.247457 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244271 4966 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:49:16.247457 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244273 4966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:49:16.247457 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244276 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:49:16.247457 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244278 4966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:49:16.247457 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244281 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:49:16.247457 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244284 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:49:16.247457 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244288 4966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:49:16.247457 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244291 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:49:16.247903 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244294 4966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:49:16.247903 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244298 4966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:49:16.247903 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244300 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:49:16.247903 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244303 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:49:16.247903 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244305 4966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:49:16.247903 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244308 4966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:49:16.247903 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244311 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:49:16.247903 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244313 4966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:49:16.247903 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244316 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:49:16.247903 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244318 4966 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:49:16.247903 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244321 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:49:16.247903 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244325 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:49:16.247903 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244328 4966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:49:16.247903 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244330 4966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:49:16.247903 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244333 4966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:49:16.247903 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244336 4966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:49:16.247903 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244338 4966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:49:16.247903 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244897 4966 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:49:16.247903 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244903 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244906 4966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244909 4966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244912 4966 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244928 4966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244932 4966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244935 4966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244938 4966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244941 4966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244944 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244947 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244949 4966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244952 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244954 4966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244957 4966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244960 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244963 4966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244966 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244968 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244971 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:49:16.248421 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244973 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244976 4966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244979 4966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244982 4966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244985 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244987 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244990 4966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244993 4966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244995 4966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.244998 4966 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245001 4966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245004 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245007 4966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245010 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245012 4966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245015 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245017 4966 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245020 4966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245022 4966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245025 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:49:16.249079 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245027 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:49:16.249991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245030 4966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:49:16.249991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245033 4966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:49:16.249991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245035 4966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:49:16.249991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245038 4966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:49:16.249991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245040 4966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:49:16.249991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245043 4966 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:49:16.249991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245046 4966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:49:16.249991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245048 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:49:16.249991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245051 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:49:16.249991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245054 4966 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:49:16.249991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245056 4966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:49:16.249991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245059 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:49:16.249991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245061 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:49:16.249991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245065 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:49:16.249991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245068 4966 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:49:16.249991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245070 4966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:49:16.249991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245075 4966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:49:16.249991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245079 4966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:49:16.249991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245082 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245085 4966 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245088 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245091 4966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245094 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245096 4966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245100 4966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245103 4966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245105 4966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245108 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245111 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245114 4966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245117 4966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245120 4966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245122 4966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245125 4966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245127 4966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245130 4966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245133 4966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245135 4966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:49:16.252213 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245138 4966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:49:16.253160 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245140 4966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:49:16.253160 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245143 4966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:49:16.253160 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245146 4966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:49:16.253160 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245149 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:49:16.253160 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245152 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:49:16.253160 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245154 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:49:16.253160 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245157 4966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:49:16.253160 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245160 4966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:49:16.253160 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245163 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:49:16.253160 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245165 4966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:49:16.253160 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245168 4966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:49:16.253160 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245172 4966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:49:16.253160 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245176 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:49:16.253160 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245178 4966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:49:16.253160 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245181 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:49:16.253160 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245184 4966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:49:16.253160 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245187 4966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:49:16.253160 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245189 4966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:49:16.253160 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245193 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245195 4966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245198 4966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245201 4966 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245205 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245207 4966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.245210 4966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245305 4966 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245312 4966 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245318 4966 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245323 4966 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245327 4966 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245331 4966 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245336 4966 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245340 4966 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245343 4966 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245346 4966 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245350 4966 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245354 4966 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245357 4966 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245360 4966 flags.go:64] FLAG: --cgroup-root="" Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245363 4966 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245366 4966 flags.go:64] FLAG: --client-ca-file="" Nov 04 03:49:16.254018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245369 4966 flags.go:64] FLAG: --cloud-provider="external" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245372 4966 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245377 4966 flags.go:64] FLAG: --cluster-domain="" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245380 4966 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245383 4966 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245386 4966 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245390 4966 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245394 4966 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245398 4966 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245401 4966 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245405 4966 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245408 4966 flags.go:64] FLAG: --contention-profiling="false" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245411 4966 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245414 4966 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245417 4966 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245420 4966 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245428 4966 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245431 4966 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245434 4966 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245437 4966 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245440 4966 flags.go:64] FLAG: --enable-server="true" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245444 4966 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245447 4966 flags.go:64] FLAG: --event-burst="100" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245451 4966 flags.go:64] FLAG: --event-qps="50" Nov 04 03:49:16.255029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245454 4966 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245457 4966 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245460 4966 flags.go:64] FLAG: --eviction-hard="" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245464 4966 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245468 4966 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245471 4966 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245474 4966 flags.go:64] FLAG: --eviction-soft="" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245477 4966 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245480 4966 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245483 4966 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245486 4966 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245490 4966 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245493 4966 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245496 4966 flags.go:64] FLAG: --feature-gates="" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245502 4966 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245505 4966 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245509 4966 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245512 4966 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245515 4966 flags.go:64] FLAG: --healthz-port="10248" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245519 4966 flags.go:64] FLAG: --help="false" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245522 4966 flags.go:64] FLAG: --hostname-override="ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245526 4966 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245529 4966 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 03:49:16.256111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245532 4966 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245536 4966 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245540 4966 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245543 4966 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245546 4966 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245549 4966 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245552 4966 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245555 4966 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245559 4966 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245562 4966 flags.go:64] FLAG: --kube-reserved="" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245565 4966 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245567 4966 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245571 4966 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245574 4966 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245577 4966 flags.go:64] FLAG: --lock-file="" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245580 4966 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245582 4966 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245586 4966 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245592 4966 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245595 4966 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245598 4966 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245601 4966 flags.go:64] FLAG: --logging-format="text" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245604 4966 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245607 4966 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 03:49:16.257109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245610 4966 flags.go:64] FLAG: --manifest-url="" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245613 4966 flags.go:64] FLAG: --manifest-url-header="" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245618 4966 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245621 4966 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245625 4966 flags.go:64] FLAG: --max-pods="110" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245636 4966 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245640 4966 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245642 4966 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245646 4966 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245650 4966 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245654 4966 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245657 4966 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245665 4966 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245670 4966 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245673 4966 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245676 4966 flags.go:64] FLAG: --pod-cidr="" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245679 4966 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245682 4966 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245685 4966 flags.go:64] FLAG: --pods-per-core="0" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245689 4966 flags.go:64] FLAG: --port="10250" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245692 4966 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245695 4966 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-08b3288f5328a15ef" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245698 4966 flags.go:64] FLAG: --qos-reserved="" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245701 4966 flags.go:64] FLAG: --read-only-port="10255" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245704 4966 flags.go:64] FLAG: --register-node="true" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245707 4966 flags.go:64] FLAG: --register-with-taints="" Nov 04 03:49:16.258175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245711 4966 flags.go:64] FLAG: --registry-burst="10" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245714 4966 flags.go:64] FLAG: --registry-qps="5" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245717 4966 flags.go:64] FLAG: --reserved-cpus="" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245720 4966 flags.go:64] FLAG: --reserved-memory="" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245724 4966 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245727 4966 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245730 4966 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245733 4966 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245736 4966 flags.go:64] FLAG: --runonce="false" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245739 4966 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245742 4966 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245746 4966 flags.go:64] FLAG: --seccomp-default="false" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245751 4966 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245754 4966 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245757 4966 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245763 4966 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245767 4966 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245770 4966 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245773 4966 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245777 4966 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245780 4966 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245784 4966 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245787 4966 flags.go:64] FLAG: --system-cgroups="" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245790 4966 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245795 4966 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 03:49:16.259153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245798 4966 flags.go:64] FLAG: --tls-cert-file="" Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245801 4966 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245806 4966 flags.go:64] FLAG: --tls-min-version="" Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245809 4966 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245812 4966 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245815 4966 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245818 4966 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245821 4966 flags.go:64] FLAG: --v="2" Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245825 4966 flags.go:64] FLAG: --version="false" Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245829 4966 flags.go:64] FLAG: --vmodule="" Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245833 4966 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.245836 4966 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246001 4966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246006 4966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246010 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246013 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246016 4966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246019 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246022 4966 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246025 4966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246033 4966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246036 4966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:49:16.260493 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246041 4966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246044 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246047 4966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246050 4966 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246054 4966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246057 4966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246060 4966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246063 4966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246065 4966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246068 4966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246071 4966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246074 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246076 4966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246079 4966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246082 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246084 4966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246087 4966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246090 4966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246093 4966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246096 4966 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:49:16.261215 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246099 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246101 4966 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246104 4966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246107 4966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246109 4966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246112 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246114 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246117 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246120 4966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246123 4966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246126 4966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246129 4966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246134 4966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246137 4966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246139 4966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246142 4966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246146 4966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246149 4966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246151 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246154 4966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:49:16.261852 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246169 4966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:49:16.262507 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246173 4966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:49:16.262507 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246175 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:49:16.262507 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246178 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:49:16.262507 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246181 4966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:49:16.262507 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246184 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:49:16.262507 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246186 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:49:16.262507 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246189 4966 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:49:16.262507 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246192 4966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:49:16.262507 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246195 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:49:16.262507 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246197 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:49:16.262507 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246200 4966 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:49:16.262507 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246203 4966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:49:16.262507 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246205 4966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:49:16.262507 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246208 4966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:49:16.262507 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246211 4966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:49:16.262507 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246213 4966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:49:16.262507 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246216 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:49:16.262507 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246219 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:49:16.262507 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246221 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:49:16.263038 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246224 4966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:49:16.263038 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246228 4966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:49:16.263038 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246231 4966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:49:16.263038 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246234 4966 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:49:16.263038 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246240 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:49:16.263038 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246243 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:49:16.263038 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246246 4966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:49:16.263038 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246249 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:49:16.263038 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246253 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:49:16.263038 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246256 4966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:49:16.263038 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246259 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:49:16.263038 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246261 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:49:16.263038 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246264 4966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:49:16.263038 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246266 4966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:49:16.263038 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246269 4966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:49:16.263038 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246272 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:49:16.263038 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246275 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:49:16.263038 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246279 4966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:49:16.263569 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246283 4966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:49:16.263569 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246287 4966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:49:16.263569 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246290 4966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:49:16.263569 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246292 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:49:16.263569 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246296 4966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:49:16.263569 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246298 4966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:49:16.263569 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246301 4966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:49:16.263569 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246304 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:49:16.263569 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246307 4966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:49:16.263569 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246309 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:49:16.263569 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246312 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:49:16.263569 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246314 4966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:49:16.263569 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246317 4966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:49:16.263569 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246320 4966 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:49:16.263569 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246322 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:49:16.263569 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246325 4966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:49:16.263569 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246328 4966 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:49:16.263569 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246331 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:49:16.264124 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.246359 4966 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 03:49:16.264124 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246405 4966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:49:16.264124 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246410 4966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:49:16.264124 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246414 4966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:49:16.264124 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246419 4966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:49:16.264124 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246422 4966 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:49:16.264124 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246424 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:49:16.264124 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246428 4966 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:49:16.264124 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246431 4966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:49:16.264124 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246433 4966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:49:16.264124 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246436 4966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:49:16.264124 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246439 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:49:16.264124 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246442 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:49:16.264124 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246445 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:49:16.264124 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246448 4966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:49:16.264124 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246451 4966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246454 4966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246457 4966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246459 4966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246462 4966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246465 4966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246468 4966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246470 4966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246473 4966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246475 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246478 4966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246481 4966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246484 4966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246487 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246489 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246492 4966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246495 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246497 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246502 4966 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246505 4966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:49:16.264647 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246508 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:49:16.265300 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246512 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:49:16.265300 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246515 4966 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:49:16.265300 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246518 4966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:49:16.265300 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246520 4966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:49:16.265300 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246523 4966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:49:16.265300 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246526 4966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:49:16.265300 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246529 4966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:49:16.265300 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246531 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:49:16.265300 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246534 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:49:16.265300 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246537 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:49:16.265300 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246540 4966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:49:16.265300 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246544 4966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:49:16.265300 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246548 4966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:49:16.265300 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246551 4966 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:49:16.265300 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246554 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:49:16.265300 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246557 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:49:16.265300 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246560 4966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:49:16.265300 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246563 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:49:16.265300 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246565 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:49:16.265828 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246568 4966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:49:16.265828 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246571 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:49:16.265828 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246573 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:49:16.265828 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246576 4966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:49:16.265828 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246579 4966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:49:16.265828 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246581 4966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:49:16.265828 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246584 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:49:16.265828 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246587 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:49:16.265828 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246591 4966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:49:16.265828 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246594 4966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:49:16.265828 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246598 4966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:49:16.265828 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246601 4966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:49:16.265828 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246604 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:49:16.265828 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246608 4966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:49:16.265828 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246611 4966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:49:16.265828 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246614 4966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:49:16.265828 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246617 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:49:16.265828 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246619 4966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:49:16.265828 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246622 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246625 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246628 4966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246630 4966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246633 4966 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246636 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246638 4966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246641 4966 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246644 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246646 4966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246649 4966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246652 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246655 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246657 4966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246660 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246663 4966 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246665 4966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246668 4966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246670 4966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246673 4966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:49:16.266367 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246676 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246679 4966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246682 4966 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246685 4966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246689 4966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246691 4966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246695 4966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246704 4966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246707 4966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246710 4966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246713 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246716 4966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246719 4966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.246721 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.251395 4966 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.251419 4966 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.251444 4966 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.251452 4966 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251535 4966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251543 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:49:16.266907 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251548 4966 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251555 4966 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251560 4966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251565 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251570 4966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251575 4966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251580 4966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251585 4966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251589 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251593 4966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251597 4966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251602 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251606 4966 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251610 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251615 4966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251619 4966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251623 4966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251627 4966 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251632 4966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251636 4966 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:49:16.267512 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251640 4966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:49:16.268093 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251645 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:49:16.268093 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251648 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:49:16.268093 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251653 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:49:16.268093 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251657 4966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:49:16.268093 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251661 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:49:16.268093 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251665 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:49:16.268093 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251670 4966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:49:16.268093 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251674 4966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:49:16.268093 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251678 4966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:49:16.268093 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251682 4966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:49:16.268093 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251686 4966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:49:16.268093 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251690 4966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:49:16.268093 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251694 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:49:16.268093 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251698 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:49:16.268093 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251702 4966 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:49:16.268093 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251706 4966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:49:16.268093 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251710 4966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:49:16.268093 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251714 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:49:16.268093 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251718 4966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:49:16.268600 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251722 4966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:49:16.268600 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251726 4966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:49:16.268600 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251730 4966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:49:16.268600 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251734 4966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:49:16.268600 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251739 4966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:49:16.268600 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251743 4966 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:49:16.268600 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251748 4966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:49:16.268600 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251752 4966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:49:16.268600 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251756 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:49:16.268600 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251761 4966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:49:16.268600 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251766 4966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:49:16.268600 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251771 4966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:49:16.268600 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251778 4966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:49:16.268600 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251783 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:49:16.268600 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251788 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:49:16.268600 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251793 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:49:16.268600 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251799 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:49:16.268600 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251803 4966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:49:16.268600 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251809 4966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:49:16.269247 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251815 4966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:49:16.269247 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251821 4966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:49:16.269247 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251826 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:49:16.269247 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251830 4966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:49:16.269247 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251835 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:49:16.269247 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251839 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:49:16.269247 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251844 4966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:49:16.269247 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251849 4966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:49:16.269247 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251853 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:49:16.269247 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251857 4966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:49:16.269247 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251861 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:49:16.269247 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251866 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:49:16.269247 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251870 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:49:16.269247 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251874 4966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:49:16.269247 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251879 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:49:16.269247 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251884 4966 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:49:16.269247 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251889 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:49:16.269247 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251893 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:49:16.269247 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251897 4966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251901 4966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251906 4966 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251910 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251914 4966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251935 4966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251939 4966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251944 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251949 4966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251955 4966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251960 4966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251964 4966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251969 4966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251973 4966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251978 4966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251982 4966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251988 4966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251992 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.251996 4966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252000 4966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:49:16.269741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252005 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:49:16.270273 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252009 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:49:16.270273 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252014 4966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:49:16.270273 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252018 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:49:16.270273 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252023 4966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:49:16.270273 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252027 4966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:49:16.270273 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252031 4966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:49:16.270273 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.252077 4966 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 03:49:16.270273 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252347 4966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:49:16.270273 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252356 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:49:16.270273 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252361 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:49:16.270273 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252367 4966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:49:16.270273 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252372 4966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:49:16.270273 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252377 4966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:49:16.270273 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252382 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:49:16.270273 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252387 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:49:16.270273 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252391 4966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252396 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252400 4966 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252405 4966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252409 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252414 4966 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252419 4966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252424 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252428 4966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252433 4966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252437 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252441 4966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252446 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252451 4966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252456 4966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252460 4966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252464 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252469 4966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252473 4966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252477 4966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:49:16.270836 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252481 4966 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:49:16.271346 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252485 4966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:49:16.271346 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252490 4966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:49:16.271346 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252494 4966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:49:16.271346 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252498 4966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:49:16.271346 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252502 4966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:49:16.271346 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252507 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:49:16.271346 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252511 4966 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:49:16.271346 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252515 4966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:49:16.271346 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252520 4966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:49:16.271346 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252524 4966 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:49:16.271346 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252528 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:49:16.271346 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252533 4966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:49:16.271346 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252539 4966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:49:16.271346 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252547 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:49:16.271346 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252552 4966 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:49:16.271346 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252557 4966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:49:16.271346 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252561 4966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:49:16.271346 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252566 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:49:16.271346 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252570 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252575 4966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252579 4966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252583 4966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252588 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252593 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252598 4966 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252603 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252607 4966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252612 4966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252616 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252620 4966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252625 4966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252629 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252633 4966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252637 4966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252642 4966 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252647 4966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252652 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252656 4966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:49:16.271956 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252660 4966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:49:16.272449 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252664 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:49:16.272449 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252669 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:49:16.272449 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252673 4966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:49:16.272449 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252677 4966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:49:16.272449 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252681 4966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:49:16.272449 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252687 4966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:49:16.272449 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252694 4966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:49:16.272449 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252698 4966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:49:16.272449 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252704 4966 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:49:16.272449 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252708 4966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:49:16.272449 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252717 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:49:16.272449 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252722 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:49:16.272449 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252726 4966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:49:16.272449 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252731 4966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:49:16.272449 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252735 4966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:49:16.272449 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252739 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:49:16.272449 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252744 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:49:16.272449 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252748 4966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:49:16.272449 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252752 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:49:16.272973 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252757 4966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:49:16.272973 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252762 4966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:49:16.272973 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252766 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:49:16.272973 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252771 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:49:16.272973 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252775 4966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:49:16.272973 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252779 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:49:16.272973 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252783 4966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:49:16.272973 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252788 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:49:16.272973 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252792 4966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:49:16.272973 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252796 4966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:49:16.272973 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252800 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:49:16.272973 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252805 4966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:49:16.272973 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252809 4966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:49:16.272973 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252814 4966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:49:16.272973 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252818 4966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:49:16.272973 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252822 4966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:49:16.272973 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252827 4966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:49:16.272973 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252831 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:49:16.272973 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252837 4966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:49:16.273473 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.252875 4966 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 03:49:16.273473 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252957 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:49:16.273473 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252965 4966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:49:16.273473 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252972 4966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:49:16.273473 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252979 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:49:16.273473 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252986 4966 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:49:16.273473 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252992 4966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:49:16.273473 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.252997 4966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:49:16.273473 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253002 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:49:16.273473 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253007 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:49:16.273473 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253011 4966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:49:16.273473 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253015 4966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:49:16.273473 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253020 4966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:49:16.273473 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253024 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:49:16.273473 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253030 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:49:16.273473 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253034 4966 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:49:16.273873 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253039 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:49:16.273873 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253043 4966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:49:16.273873 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253048 4966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:49:16.273873 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253053 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:49:16.273873 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253057 4966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:49:16.273873 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253061 4966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:49:16.273873 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253066 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:49:16.273873 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253070 4966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:49:16.273873 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253074 4966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:49:16.273873 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253079 4966 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:49:16.273873 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253084 4966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:49:16.273873 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253088 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:49:16.273873 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253093 4966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:49:16.273873 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253097 4966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:49:16.273873 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253102 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:49:16.273873 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253109 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:49:16.273873 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253114 4966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:49:16.273873 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253118 4966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:49:16.273873 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253122 4966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:49:16.274390 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253129 4966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:49:16.274390 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253135 4966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:49:16.274390 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253141 4966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:49:16.274390 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253146 4966 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:49:16.274390 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253150 4966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:49:16.274390 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253154 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:49:16.274390 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253159 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:49:16.274390 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253163 4966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:49:16.274390 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253167 4966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:49:16.274390 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253171 4966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:49:16.274390 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253174 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:49:16.274390 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253179 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:49:16.274390 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253184 4966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:49:16.274390 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253188 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:49:16.274390 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253193 4966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:49:16.274390 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253197 4966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:49:16.274390 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253202 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:49:16.274390 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253207 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:49:16.274390 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253211 4966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253216 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253220 4966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253225 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253229 4966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253234 4966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253237 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253243 4966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253248 4966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253252 4966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253259 4966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253264 4966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253268 4966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253273 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253277 4966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253282 4966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253287 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253292 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253297 4966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253301 4966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:49:16.274859 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253306 4966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253310 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253314 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253319 4966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253323 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253328 4966 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253332 4966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253337 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253341 4966 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253345 4966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253350 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253354 4966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253359 4966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253363 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253367 4966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253372 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253377 4966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253381 4966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253385 4966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253390 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:49:16.275401 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253394 4966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:49:16.275889 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253398 4966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:49:16.275889 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253405 4966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:49:16.275889 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253410 4966 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:49:16.275889 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253414 4966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:49:16.275889 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253419 4966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:49:16.275889 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253423 4966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:49:16.275889 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253427 4966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:49:16.275889 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253433 4966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:49:16.275889 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253438 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:49:16.275889 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253442 4966 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:49:16.275889 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253446 4966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:49:16.275889 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253453 4966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:49:16.275889 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:16.253458 4966 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:49:16.275889 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.253549 4966 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 03:49:16.275889 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.255131 4966 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 04 03:49:16.275889 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.255209 4966 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 04 03:49:16.275889 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.255479 4966 server.go:1014] "Starting client certificate rotation" Nov 04 03:49:16.275889 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.255604 4966 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 03:49:16.276344 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.255658 4966 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 03:43:35 +0000 UTC" deadline="2028-06-01 14:21:21.934465623 +0000 UTC" Nov 04 03:49:16.276344 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.255695 4966 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13810h32m5.678772103s" Nov 04 03:49:16.276344 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.256288 4966 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 03:49:16.276344 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.258089 4966 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 03:49:16.276344 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.261720 4966 log.go:25] "Validated CRI v1 runtime API" Nov 04 03:49:16.276344 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.263786 4966 log.go:25] "Validated CRI v1 image API" Nov 04 03:49:16.276344 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.264691 4966 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 03:49:16.276344 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.267521 4966 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9cf85cde-b97e-42df-94bc-7570fe38d0f5:/dev/nvme0n1p3 bfe504cc-bf8b-4943-b41e-9f868b7133ca:/dev/nvme0n1p4] Nov 04 03:49:16.276599 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.267544 4966 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02ac35b4cb7404a4fb3d5ae493b982826d38cb4f4ee19f82583104436f253f62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02ac35b4cb7404a4fb3d5ae493b982826d38cb4f4ee19f82583104436f253f62/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/053490f48702475ba5ceeba97610c47fbb3bd28090536d601c3ca63e156b8b34/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/053490f48702475ba5ceeba97610c47fbb3bd28090536d601c3ca63e156b8b34/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0991dbab3a77a90594ed04c9b3281fe729c4d76ca82aff5cd740d5937bc7219e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0991dbab3a77a90594ed04c9b3281fe729c4d76ca82aff5cd740d5937bc7219e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/09df2a931fa2c83c00586babb549bfbbe214e7590b4938a62dbfbc2838349918/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/09df2a931fa2c83c00586babb549bfbbe214e7590b4938a62dbfbc2838349918/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a5f14aae34c9231e88c3a57c9892d6a604279f39f7f89a165f67896f845fc8a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a5f14aae34c9231e88c3a57c9892d6a604279f39f7f89a165f67896f845fc8a/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26e9731be30c71c75b18565bad3d98e6c1838b88918eb12c8f0bb754feca305a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26e9731be30c71c75b18565bad3d98e6c1838b88918eb12c8f0bb754feca305a/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/342e443435b836d0cf9df45f36d28198960585e34a612e38b5283e83b250beab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/342e443435b836d0cf9df45f36d28198960585e34a612e38b5283e83b250beab/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/435aab2da18b8024da0c2fdd7672a6661ac9dac1fd21dd6093a7c50323ae4c16/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/435aab2da18b8024da0c2fdd7672a6661ac9dac1fd21dd6093a7c50323ae4c16/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43d5131a19d1a624b54ac101f804cda981ed9633089b31b169023d5bb60395b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43d5131a19d1a624b54ac101f804cda981ed9633089b31b169023d5bb60395b7/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/571da96cb8b6a018b0047426b1c5561018ac9f326cf4ad56f325132b4fcbec4b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/571da96cb8b6a018b0047426b1c5561018ac9f326cf4ad56f325132b4fcbec4b/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/747fe1cc8b3517919d67735d1b24ebf8bc097f87f185660ed9f0c0bd64c45773/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/747fe1cc8b3517919d67735d1b24ebf8bc097f87f185660ed9f0c0bd64c45773/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/75dd49f74d6e64dd95776fe7d6053b7f8010387a3ac35b7c7413713dbc41ba5e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/75dd49f74d6e64dd95776fe7d6053b7f8010387a3ac35b7c7413713dbc41ba5e/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ddf93747976c4469160b51e7421723397597f92296ee22a6001853599bd527b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ddf93747976c4469160b51e7421723397597f92296ee22a6001853599bd527b/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/908140abe84c9bde75743f0962814b626593b42e67045aee092d5be0fe496d71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/908140abe84c9bde75743f0962814b626593b42e67045aee092d5be0fe496d71/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/918686dc6f7e96901c7733bc446a40b812cd1f44b39515659d8bdc3e61b526b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/918686dc6f7e96901c7733bc446a40b812cd1f44b39515659d8bdc3e61b526b8/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93905c9baf281e2ae8e958a45a501766968cca88dcc1f69544fb23f30e845554/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93905c9baf281e2ae8e958a45a501766968cca88dcc1f69544fb23f30e845554/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97c7fe96419bb1a3c65f7184da334ca0636c3182e80f416470a8a21bf6c41b61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97c7fe96419bb1a3c65f7184da334ca0636c3182e80f416470a8a21bf6c41b61/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a26a5b00ae7f18543adc8f26e387393e0fbb02a58d5c7f5ed524ffad17eb4a4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a26a5b00ae7f18543adc8f26e387393e0fbb02a58d5c7f5ed524ffad17eb4a4d/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd4f620630a7a6a3d7560eff980ce5bd4108983f766deef2342e0541d416e3b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd4f620630a7a6a3d7560eff980ce5bd4108983f766deef2342e0541d416e3b1/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c56d598b430b5f4e609403bdd5c77a25b76bb388cabbe0671da260a43c084ec8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c56d598b430b5f4e609403bdd5c77a25b76bb388cabbe0671da260a43c084ec8/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd27c2ebd27cf211c4d38ad9efc9a443a46483c9bc3aec8bdc605979b7ab4467/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd27c2ebd27cf211c4d38ad9efc9a443a46483c9bc3aec8bdc605979b7ab4467/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e47ff45ec5397b88dafbf3a2734ad5b34db3d4d6e13746a247bb5c3e9cae99e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e47ff45ec5397b88dafbf3a2734ad5b34db3d4d6e13746a247bb5c3e9cae99e3/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea290fb0d10aeea5436aeedfbcd14d5e027d56b395c4f74388f7269f1b8542cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea290fb0d10aeea5436aeedfbcd14d5e027d56b395c4f74388f7269f1b8542cc/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd59a627fdda9658fab68cef4d1850c1ae343b40131951cc6eebdbea3cc3f0ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd59a627fdda9658fab68cef4d1850c1ae343b40131951cc6eebdbea3cc3f0ab/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0df87f5d-31f4-4571-82b6-fe75c434ff27/volumes/kubernetes.io~projected/kube-api-access-w9fvw:{mountpoint:/var/lib/kubelet/pods/0df87f5d-31f4-4571-82b6-fe75c434ff27/volumes/kubernetes.io~projected/kube-api-access-w9fvw major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19970c28-9dee-4377-8e98-1fc65473d3e1/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/19970c28-9dee-4377-8e98-1fc65473d3e1/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19970c28-9dee-4377-8e98-1fc65473d3e1/volumes/kubernetes.io~projected/kube-api-access-n8q89:{mountpoint:/var/lib/kubelet/pods/19970c28-9dee-4377-8e98-1fc65473d3e1/volumes/kubernetes.io~projected/kube-api-access-n8q89 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19970c28-9dee-4377-8e98-1fc65473d3e1/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/19970c28-9dee-4377-8e98-1fc65473d3e1/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57/volumes/kubernetes.io~projected/kube-api-access-knwrb:{mountpoint:/var/lib/kubelet/pods/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57/volumes/kubernetes.io~projected/kube-api-access-knwrb major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57/volumes/kubernetes.io~secret/default-certificate major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57/volumes/kubernetes.io~secret/stats-auth major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c2ac1f5-6e08-48e3-ac83-a55f9181f4be/volumes/kubernetes.io~projected/kube-api-access-zhvtl:{mountpoint:/var/lib/kubelet/pods/1c2ac1f5-6e08-48e3-ac83-a55f9181f4be/volumes/kubernetes.io~projected/kube-api-access-zhvtl major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f0a14d4-9771-4aa2-a6bd-87f5f73ee931/volumes/kubernetes.io~projected/kube-api-access-sf4rk:{mountpoint:/var/lib/kubelet/pods/1f0a14d4-9771-4aa2-a6bd-87f5f73ee931/volumes/kubernetes.io~projected/kube-api-access-sf4rk major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21ac7748-5e3d-427e-bd15-93cf4754cb0e/volumes/kubernetes.io~projected/kube-api-access-5rpkk:{mountpoint:/var/lib/kubelet/pods/21ac7748-5e3d-427e-bd15-93cf4754cb0e/volumes/kubernetes.io~projected/kube-api-access-5rpkk major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26bb99fd-d223-4642-b147-b22bebd00835/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/26bb99fd-d223-4642-b147-b22bebd00835/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27d72728-5623-41e0-bee5-a09937f9f87a/volumes/kubernetes.io~projected/kube-api-access-kr88g:{mountpoint:/var/lib/kubelet/pods/27d72728-5623-41e0-bee5-a09937f9f87a/volumes/kubernetes.io~projected/kube-api-access-kr88g major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d275592-23f8-4edc-aff7-016930318629/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/2d275592-23f8-4edc-aff7-016930318629/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d275592-23f8-4edc-aff7-016930318629/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/2d275592-23f8-4edc-aff7-016930318629/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d275592-23f8-4edc-aff7-016930318629/volumes/kubernetes.io~projected/kube-api-access-kgv6q:{mountpoint:/var/lib/kubelet/pods/2d275592-23f8-4edc-aff7-016930318629/volumes/kubernetes.io~projected/kube-api-access-kgv6q major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30f18911-ab5a-44e3-88b2-6508325ceb39/volumes/kubernetes.io~projected/kube-api-access-q5t7v:{mountpoint:/var/lib/kubelet/pods/30f18911-ab5a-44e3-88b2-6508325ceb39/volumes/kubernetes.io~projected/kube-api-access-q5t7v major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3/volumes/kubernetes.io~projected/kube-api-access-lsvdh:{mountpoint:/var/lib/kubelet/pods/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3/volumes/kubernetes.io~projected/kube-api-access-lsvdh major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/363d8742-c0a7-405c-9683-7b12afbd8bcc/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/363d8742-c0a7-405c-9683-7b12afbd8bcc/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/363d8742-c0a7-405c-9683-7b12afbd8bcc/volumes/kubernetes.io~projected/kube-api-access-952rw:{mountpoint:/var/lib/kubelet/pods/363d8742-c0a7-405c-9683-7b12afbd8bcc/volumes/kubernetes.io~projected/kube-api-access-952rw major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/363d8742-c0a7-405c-9683-7b12afbd8bcc/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/363d8742-c0a7-405c-9683-7b12afbd8bcc/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/363d8742-c0a7-405c-9683-7b12afbd8bcc/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/363d8742-c0a7-405c-9683-7b12afbd8bcc/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/366aeead-59de-4180-8d14-e8b821d04e22/volumes/kubernetes.io~projected/kube-api-access-8852w:{mountpoint:/var/lib/kubelet/pods/366aeead-59de-4180-8d14-e8b821d04e22/volumes/kubernetes.io~projected/kube-api-access-8852w major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/366aeead-59de-4180-8d14-e8b821d04e22/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/366aeead-59de-4180-8d14-e8b821d04e22/volumes/kubernetes.io~secret/signing-key major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391e5382-7119-4085-9939-cd81fc8b2d73/volumes/kubernetes.io~projected/kube-api-access-mq2fj:{mountpoint:/var/lib/kubelet/pods/391e5382-7119-4085-9939-cd81fc8b2d73/volumes/kubernetes.io~projected/kube-api-access-mq2fj major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b456f4f-071a-4e5f-b8b6-51c3ee790656/volumes/kubernetes.io~projected/kube-api-access-s6hlp:{mountpoint:/var/lib/kubelet/pods/3b456f4f-071a-4e5f-b8b6-51c3ee790656/volumes/kubernetes.io~projected/kube-api-access-s6hlp major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b456f4f-071a-4e5f-b8b6-51c3ee790656/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3b456f4f-071a-4e5f-b8b6-51c3ee790656/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ce83f95-758c-4fd2-9678-441fe4309da7/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/3ce83f95-758c-4fd2-9678-441fe4309da7/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ce83f95-758c-4fd2-9678-441fe4309da7/volumes/kubernetes.io~projected/kube-api-access-sqp55:{mountpoint:/var/lib/kubelet/pods/3ce83f95-758c-4fd2-9678-441fe4309da7/volumes/kubernetes.io~projected/kube-api-access-sqp55 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ce83f95-758c-4fd2-9678-441fe4309da7/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/3ce83f95-758c-4fd2-9678-441fe4309da7/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ce83f95-758c-4fd2-9678-441fe4309da7/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/3ce83f95-758c-4fd2-9678-441fe4309da7/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d313ca1-e640-42d9-a9f7-5e3e8918dc77/volumes/kubernetes.io~projected/kube-api-access-rw88q:{mountpoint:/var/lib/kubelet/pods/3d313ca1-e640-42d9-a9f7-5e3e8918dc77/volumes/kubernetes.io~projected/kube-api-access-rw88q major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d313ca1-e640-42d9-a9f7-5e3e8918dc77/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3d313ca1-e640-42d9-a9f7-5e3e8918dc77/volumes/kubernetes.io~secret/serving-cert major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fbef9e3-d5de-4a35-8745-2b1dc357c52b/volumes/kubernetes.io~projected/kube-api-access-rjgp4:{mountpoint:/var/lib/kubelet/pods/3fbef9e3-d5de-4a35-8745-2b1dc357c52b/volumes/kubernetes.io~projected/kube-api-access-rjgp4 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7714ee23-de1d-4e5d-88ba-cce91e08ba23/volumes/kubernetes.io~projected/kube-api-access-fb5w2:{mountpoint:/var/lib/kubelet/pods/7714ee23-de1d-4e5d-88ba-cce91e08ba23/volumes/kubernetes.io~projected/kube-api-access-fb5w2 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78e55dc9-354d-455f-9d02-b3cfd5885d65/volumes/kubernetes.io~projected/kube-api-access-6w8m8:{mountpoint:/var/lib/kubelet/pods/78e55dc9-354d-455f-9d02-b3cfd5885d65/volumes/kubernetes.io~projected/kube-api-access-6w8m8 major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/795d3139-8d19-4852-bd9c-3c60b01a97f6/volumes/kubernetes.io~projected/kube-api-access-wnklb:{mountpoint:/var/lib/kubelet/pods/795d3139-8d19-4852-bd9c-3c60b01a97f6/volumes/kubernetes.io~projected/kube-api-access-wnklb major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b632ca9-a695-450e-a1a9-b5357ebf4d7d/volumes/kubernetes.io~projected/kube-api-access-szjzl:{mountpoint:/var/lib/kubelet/pods/7b632ca9-a695-450e-a1a9-b5357ebf4d7d/volumes/kubernetes.io~projected/kube-api-access-szjzl major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93149be5-0d1f-4b08-b33e-d2e43bac726c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/93149be5-0d1f-4b08-b33e-d2e43bac726c/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ab83f7b-3396-4cc7-875e-c2cc62696ff5/volumes/kubernetes.io~projected/kube-api-access-4vf5x:{mountpoint:/var/lib/kubelet/pods/9ab83f7b-3396-4cc7-875e-c2cc62696ff5/volumes/kubernetes.io~projected/kube-api-access-4vf5x major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/volumes/kubernetes.io~projected/kube-api-access-zvg2z:{mountpoint:/var/lib/kubelet/pods/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/volumes/kubernetes.io~projected/kube-api-access-zvg2z major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b/volumes/kubernetes.io~projected/kube-api-access-6dcsj:{mountpoint:/var/lib/kubelet/pods/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b/volumes/kubernetes.io~projected/kube-api-access-6dcsj major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcc0d689-48b6-467e-bdd0-6a03878a52b6/volumes/kubernetes.io~projected/kube-api-access-hjrzz:{mountpoint:/var/lib/kubelet/pods/bcc0d689-48b6-467e-bdd0-6a03878a52b6/volumes/kubernetes.io~projected/kube-api-access-hjrzz major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcc0d689-48b6-467e-bdd0-6a03878a52b6/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/bcc0d689-48b6-467e-bdd0-6a03878a52b6/volumes/kubernetes.io~secret/cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c12382e2-90c4-45bf-bdd8-27653073cc5d/volumes/kubernetes.io~projected/kube-api-access-lj5tf:{mountpoint:/var/lib/kubelet/pods/c12382e2-90c4-45bf-bdd8-27653073cc5d/volumes/kubernetes.io~projected/kube-api-access-lj5tf major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2052e7f-5ee8-4066-bb29-cd227fa0965b/volumes/kubernetes.io~projected/kube-api-access-pcjkb:{mountpoint:/var/lib/kubelet/pods/d2052e7f-5ee8-4066-bb29-cd227fa0965b/volumes/kubernetes.io~projected/kube-api-access-pcjkb major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2052e7f-5ee8-4066-bb29-cd227fa0965b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/d2052e7f-5ee8-4066-bb29-cd227fa0965b/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d25872c4-1eee-48b8-a4be-9384dae267ec/volumes/kubernetes.io~projected/kube-api-access-dfk6m:{mountpoint:/var/lib/kubelet/pods/d25872c4-1eee-48b8-a4be-9384dae267ec/volumes/kubernetes.io~projected/kube-api-access-dfk6m major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7a99c19-45c9-4294-b2af-42d20306ff28/volumes/kubernetes.io~projected/kube-api-access-w7b4p:{mountpoint:/var/lib/kubelet/pods/f7a99c19-45c9-4294-b2af-42d20306ff28/volumes/kubernetes.io~projected/kube-api-access-w7b4p major:0 minor:60 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/155c869d117539a461ece47c46364986785ac52fe15c8fa8c78a5bf0aa8680eb/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/4a4d23512d91e2e0dd450ddd5771499885d78d9516032d7ccd22fad91773fe64/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/11c166218269bcfdbb975a5fd9d974b871fb806327db454da614c4d5058962a6/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/fb70b9461739555cf60793a7332706b5b09eaeb2e97306811c7c739d1b5c5862/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d26a19c0a5f669657e90e1e6239b67bab14bd2aa1b2f1bdd466111773e274b9a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/34c60ad25ff3589f0972911a8c04a81e483dc9effcf60771e490c849aa241f62/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/422289313ff49072cf47b78180530aade5275ce918464050b9a5f0b101326a3f/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e67030f58ca01384071f0ab9ea98164a990aef93834006565504cec086313072/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d03d7c857ae7f9c473a8df546fd4cb5079740d0f52caa8c023a332b7f765f6e0/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/dff845051f11ae510dc63da837f2da922c54cad7401c4931fb1fa638b872e032/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/0c9e7dfc91bd74d5ac28a604acdcc7db1226de8ba7c63967399a389800d82f64/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/f3220ed45c9474ec437ec2aee141175f53ef3809b683a5a8388a1930d89867fb/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e5b2a32428bf020bbb9c31526dc98503b1367832f090b40a43be9db94aed956d/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/f86dd55ba3e9977c1b657ed57922f1427ef769e6a37a204c4db45e6e41892e34/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/071730d3ff53795fb340bf729733b10f56aa3497fe1a506d467381986e6cb13f/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/5dacfcd8efcf399a210fdf6e676ba20b3c03f1f267976676910595dc3ec7ce51/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/d4d4efdd476346d716aa6987243a3ba8b50d359fdc119a6378aff9ebb6f77435/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/383a8d556af657766375aaa0ddc2b1cfd11c51622d942cd988ed0c5aa5d5bf33/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/203e70bdc54d501b31f66dbbea3be5fbc44083d343fd88ea3aa9badb8c06bf14/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/7d2e2fe2016ee9d1a2c4fdf1295c9f760d831e58509e031d5dcd64f17b2e5242/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d2ee2509fd754d02ba5522cae4f2749036f8624820719b992c06d4c24c629c2e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/d03e6b8ccfbd4455c91f1a270848dc028142112775a833dd8ed58d0fc0143602/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/f68efd31bd4f2fc4a3416ff6691923fc5f0e47f9dd66f300a3ccb7e3584c30a7/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/81ff929174573c1a2d86919d951e70b6d3b5d877f75dd8280d4b1f875875037c/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/2ea72e3ab9238c68cda8b90e0aeeaf6de53fce74aad8adc5861a4f125d283bb7/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/bcc3f57f1e76343b363fff4bfc7487eec30597eb3d1842fc21000d7e6b3d94f4/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/101456b218785c9d82c117aa7c6402bcee8b90e865b6781ee579b8a674d52a7d/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/10e461632669bd8af523c63d75ee900899a23a9d5998c7f9958b7404b54102ff/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/6b6d978832512eb64eb72c152d9c91ddd143a3e98de6bda6e7c55c6182972229/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/6a8296cc85ec170ba426b0925d9c395ef0d982ac1a80eafd63247a22c72eb08d/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/5b46c26824d820fb123a44a67a1ad4ca934770419b67ad655ab988249282c4da/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/2fa317021cac0a5cbeede130708ce7f046d5d10dd54f3d3c55fc97d6facc958a/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/dc6c5117a4912d9d9253f7b7b22c795d597eab4def57dab901989dbf0282e040/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/14f0c432dc7c09056087c12d733e6743adeed466ccda44857759f231c23118f2/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/a73ceda08a622aeb6e454dc9a7a14894f77c68b4a939ae36e2b1c43c7319664a/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/ffe091013d3d53d7884edac421747d4f8e4a46855b8eb945db498e87d2c69b52/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/9e1ecc81fdd8785479034b81e45f0dd67fcd51641c5c3f81e402f76466e6506c/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/adf6dec39740124befc440f0d08d102e0e628c7d93615d53b25b9656568668d6/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/ad28bf7b4308ce11d256232fe63716aa15bf6f368b5cb6d11353b3fce5e322ba/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/d4117a9f1d71c9e66cbc655fa90bf06826cd04eaee15297e675d73cce194eb85/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/3649e04dffc36f573cecc8c28747afd6a5fade9c3a8ec261e961ca0bfa45f0be/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/4ace7f5e491ce9e885c7931ec5ca2f471c5d800e8ffcd9ba60b773df693197f4/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/cc91c35a2a152de27ec663ad2ce822870053492cec3fd39e9aaffa6b0c8b2198/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/225d04b0ba1eeefd3b85cab1b02bcacc36e2e65c4a86af03ea689c28fe6f27c6/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/571677fcb41368803e6679105c9192649bd6ea007361735cb9b65bfb3cb3e50f/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/622d230e5f33ca0be4f313625e1ebd20a8f6515e99397005357074d2d6d44e02/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/3d50bb7e71805c0aba81791d40d5a62fb19a2922640f5c4850ec3c8ee4b3c3b9/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/79911188a13c358a38220775c4dd91c8c944ab4d377bcdfa5b019782006b510b/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/4926b845c8288b2bea4615fe802ea54361fdeab27f7ad9f413048cd5a375027c/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/dafe37072232b9253ccde78763e65af02c2ac92a01daa93edca0c303f607c297/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ccf134f2bea7ec05344a5eac04507c19ea848a6cec37d1315f9f3f12a0ad18fb/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/2a44bf2cd5b844f36b8b878250b4a08aac9430231cdb943f84a0a193af3781c7/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/88fd141d909192461675321930e250983dd7a7a3834338f2681070a52e5a8f04/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/8a78e0f62ca32446e8fe18e63bc9ce89d405af13b72e507bfa9ac69ef0106552/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/1e8199d8fba264476af6c98d22c63e3c7ab935dc20c1ed4f47ffbc79eba966ad/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/391805283b1497c80aa27f2329851c1353416bcab4094a1e89c77588a7cc2346/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/812e40a0c13f9b4b1e59bc4f7ecd12b535f383318beae4150b94d3efa1aebd62/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/ceb977ac86f82699eb73057c3302f2deabc2b8ba84aa6d4a7424de67c35645b8/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 04 03:49:16.278880 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.278288 4966 manager.go:217] Machine: {Timestamp:2026-11-04 03:49:16.277460937 +0000 UTC m=+0.078406757 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094272 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec219d89ae6f36ba96bceaa767421753 SystemUUID:ec219d89-ae6f-36ba-96bc-eaa767421753 BootID:0ca5310f-37da-482c-88d9-e2016640dff1 Filesystems:[{Device:/run/containers/storage/overlay-containers/918686dc6f7e96901c7733bc446a40b812cd1f44b39515659d8bdc3e61b526b8/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d2052e7f-5ee8-4066-bb29-cd227fa0965b/volumes/kubernetes.io~projected/kube-api-access-pcjkb DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7b632ca9-a695-450e-a1a9-b5357ebf4d7d/volumes/kubernetes.io~projected/kube-api-access-szjzl DeviceMajor:0 DeviceMinor:220 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd4f620630a7a6a3d7560eff980ce5bd4108983f766deef2342e0541d416e3b1/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19970c28-9dee-4377-8e98-1fc65473d3e1/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/363d8742-c0a7-405c-9683-7b12afbd8bcc/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43d5131a19d1a624b54ac101f804cda981ed9633089b31b169023d5bb60395b7/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/volumes/kubernetes.io~projected/kube-api-access-zvg2z DeviceMajor:0 DeviceMinor:227 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7714ee23-de1d-4e5d-88ba-cce91e08ba23/volumes/kubernetes.io~projected/kube-api-access-fb5w2 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ce83f95-758c-4fd2-9678-441fe4309da7/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02ac35b4cb7404a4fb3d5ae493b982826d38cb4f4ee19f82583104436f253f62/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a5f14aae34c9231e88c3a57c9892d6a604279f39f7f89a165f67896f845fc8a/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e47ff45ec5397b88dafbf3a2734ad5b34db3d4d6e13746a247bb5c3e9cae99e3/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c56d598b430b5f4e609403bdd5c77a25b76bb388cabbe0671da260a43c084ec8/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/747fe1cc8b3517919d67735d1b24ebf8bc097f87f185660ed9f0c0bd64c45773/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d313ca1-e640-42d9-a9f7-5e3e8918dc77/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/26bb99fd-d223-4642-b147-b22bebd00835/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:293 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/366aeead-59de-4180-8d14-e8b821d04e22/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:342 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3/volumes/kubernetes.io~projected/kube-api-access-lsvdh DeviceMajor:0 DeviceMinor:333 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0991dbab3a77a90594ed04c9b3281fe729c4d76ca82aff5cd740d5937bc7219e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2d275592-23f8-4edc-aff7-016930318629/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78e55dc9-354d-455f-9d02-b3cfd5885d65/volumes/kubernetes.io~projected/kube-api-access-6w8m8 DeviceMajor:0 DeviceMinor:332 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57/volumes/kubernetes.io~projected/kube-api-access-knwrb DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3fbef9e3-d5de-4a35-8745-2b1dc357c52b/volumes/kubernetes.io~projected/kube-api-access-rjgp4 DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/21ac7748-5e3d-427e-bd15-93cf4754cb0e/volumes/kubernetes.io~projected/kube-api-access-5rpkk DeviceMajor:0 DeviceMinor:222 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/908140abe84c9bde75743f0962814b626593b42e67045aee092d5be0fe496d71/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd27c2ebd27cf211c4d38ad9efc9a443a46483c9bc3aec8bdc605979b7ab4467/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea290fb0d10aeea5436aeedfbcd14d5e027d56b395c4f74388f7269f1b8542cc/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/391e5382-7119-4085-9939-cd81fc8b2d73/volumes/kubernetes.io~projected/kube-api-access-mq2fj DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c2ac1f5-6e08-48e3-ac83-a55f9181f4be/volumes/kubernetes.io~projected/kube-api-access-zhvtl DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/27d72728-5623-41e0-bee5-a09937f9f87a/volumes/kubernetes.io~projected/kube-api-access-kr88g DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97c7fe96419bb1a3c65f7184da334ca0636c3182e80f416470a8a21bf6c41b61/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9ab83f7b-3396-4cc7-875e-c2cc62696ff5/volumes/kubernetes.io~projected/kube-api-access-4vf5x DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d2052e7f-5ee8-4066-bb29-cd227fa0965b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/09df2a931fa2c83c00586babb549bfbbe214e7590b4938a62dbfbc2838349918/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a26a5b00ae7f18543adc8f26e387393e0fbb02a58d5c7f5ed524ffad17eb4a4d/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b456f4f-071a-4e5f-b8b6-51c3ee790656/volumes/kubernetes.io~projected/kube-api-access-s6hlp DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/363d8742-c0a7-405c-9683-7b12afbd8bcc/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d275592-23f8-4edc-aff7-016930318629/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/19970c28-9dee-4377-8e98-1fc65473d3e1/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/795d3139-8d19-4852-bd9c-3c60b01a97f6/volumes/kubernetes.io~projected/kube-api-access-wnklb DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/19970c28-9dee-4377-8e98-1fc65473d3e1/volumes/kubernetes.io~projected/kube-api-access-n8q89 DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/435aab2da18b8024da0c2fdd7672a6661ac9dac1fd21dd6093a7c50323ae4c16/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f0a14d4-9771-4aa2-a6bd-87f5f73ee931/volumes/kubernetes.io~projected/kube-api-access-sf4rk DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ce83f95-758c-4fd2-9678-441fe4309da7/volumes/kubernetes.io~projected/kube-api-access-sqp55 DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/053490f48702475ba5ceeba97610c47fbb3bd28090536d601c3ca63e156b8b34/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/363d8742-c0a7-405c-9683-7b12afbd8bcc/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26e9731be30c71c75b18565bad3d98e6c1838b88918eb12c8f0bb754feca305a/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93149be5-0d1f-4b08-b33e-d2e43bac726c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0df87f5d-31f4-4571-82b6-fe75c434ff27/volumes/kubernetes.io~projected/kube-api-access-w9fvw DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/363d8742-c0a7-405c-9683-7b12afbd8bcc/volumes/kubernetes.io~projected/kube-api-access-952rw DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93905c9baf281e2ae8e958a45a501766968cca88dcc1f69544fb23f30e845554/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/30f18911-ab5a-44e3-88b2-6508325ceb39/volumes/kubernetes.io~projected/kube-api-access-q5t7v DeviceMajor:0 DeviceMinor:226 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/571da96cb8b6a018b0047426b1c5561018ac9f326cf4ad56f325132b4fcbec4b/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/366aeead-59de-4180-8d14-e8b821d04e22/volumes/kubernetes.io~projected/kube-api-access-8852w DeviceMajor:0 DeviceMinor:343 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d25872c4-1eee-48b8-a4be-9384dae267ec/volumes/kubernetes.io~projected/kube-api-access-dfk6m DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b/volumes/kubernetes.io~projected/kube-api-access-6dcsj DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd59a627fdda9658fab68cef4d1850c1ae343b40131951cc6eebdbea3cc3f0ab/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ddf93747976c4469160b51e7421723397597f92296ee22a6001853599bd527b/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/3ce83f95-758c-4fd2-9678-441fe4309da7/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:221 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7a99c19-45c9-4294-b2af-42d20306ff28/volumes/kubernetes.io~projected/kube-api-access-w7b4p DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ce83f95-758c-4fd2-9678-441fe4309da7/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/342e443435b836d0cf9df45f36d28198960585e34a612e38b5283e83b250beab/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3b456f4f-071a-4e5f-b8b6-51c3ee790656/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bcc0d689-48b6-467e-bdd0-6a03878a52b6/volumes/kubernetes.io~projected/kube-api-access-hjrzz DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3d313ca1-e640-42d9-a9f7-5e3e8918dc77/volumes/kubernetes.io~projected/kube-api-access-rw88q DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d275592-23f8-4edc-aff7-016930318629/volumes/kubernetes.io~projected/kube-api-access-kgv6q DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c12382e2-90c4-45bf-bdd8-27653073cc5d/volumes/kubernetes.io~projected/kube-api-access-lj5tf DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bcc0d689-48b6-467e-bdd0-6a03878a52b6/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/75dd49f74d6e64dd95776fe7d6053b7f8010387a3ac35b7c7413713dbc41ba5e/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:053490f48702475 MacAddress:66:a4:68:72:2e:d3 Speed:10000 Mtu:8901} {Name:09df2a931fa2c83 MacAddress:36:5c:de:89:d9:c0 Speed:10000 Mtu:8901} {Name:26e9731be30c71c MacAddress:3e:63:35:1f:e9:0a Speed:10000 Mtu:8901} {Name:43d5131a19d1a62 MacAddress:46:dc:fe:5f:d1:5e Speed:10000 Mtu:8901} {Name:571da96cb8b6a01 MacAddress:12:fb:c7:f5:45:29 Speed:10000 Mtu:8901} {Name:747fe1cc8b35179 MacAddress:5a:36:f1:8d:68:13 Speed:10000 Mtu:8901} {Name:75dd49f74d6e64d MacAddress:5a:58:f9:80:b1:38 Speed:10000 Mtu:8901} {Name:8ddf93747976c44 MacAddress:aa:eb:48:40:d8:2c Speed:10000 Mtu:8901} {Name:97c7fe96419bb1a MacAddress:aa:d2:dd:91:37:66 Speed:10000 Mtu:8901} {Name:bd4f620630a7a6a MacAddress:fa:b8:f2:17:d7:23 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f9:01:2f:19 Speed:0 Mtu:9001} {Name:br-int MacAddress:3a:84:e5:7c:4a:c3 Speed:0 Mtu:8901} {Name:c56d598b430b5f4 MacAddress:a2:a0:bc:d8:6b:15 Speed:10000 Mtu:8901} {Name:cd27c2ebd27cf21 MacAddress:82:ec:a2:41:74:49 Speed:10000 Mtu:8901} {Name:e47ff45ec5397b8 MacAddress:86:21:21:02:02:e5 Speed:10000 Mtu:8901} {Name:ea290fb0d10aeea MacAddress:de:82:bb:5a:e1:3c Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f9:01:2f:19 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:46:7c:a5:96:37:cc Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:56:4a:65:c3:5b:9d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 03:49:16.279102 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.278886 4966 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 03:49:16.279102 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.278957 4966 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 03:49:16.279276 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.279254 4966 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 03:49:16.279409 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.279277 4966 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-9.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 03:49:16.279451 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.279411 4966 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 03:49:16.279451 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.279420 4966 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 03:49:16.279451 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.279434 4966 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 03:49:16.279534 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.279452 4966 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 03:49:16.279534 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.279473 4966 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 03:49:16.279652 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.279643 4966 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 03:49:16.279769 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.279759 4966 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 03:49:16.279827 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.279819 4966 kubelet.go:493] "Attempting to sync node with API server" Nov 04 03:49:16.279861 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.279832 4966 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 03:49:16.279861 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.279842 4966 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 03:49:16.279861 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.279851 4966 kubelet.go:405] "Adding apiserver pod source" Nov 04 03:49:16.279861 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.279860 4966 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 03:49:16.280816 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.280800 4966 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 03:49:16.280901 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.280828 4966 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 03:49:16.281488 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.281448 4966 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 03:49:16.285102 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.285083 4966 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 03:49:16.285102 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.285100 4966 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 03:49:16.285246 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.285208 4966 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 04 03:49:16.285488 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.285466 4966 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 03:49:16.285541 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.285506 4966 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 03:49:16.285619 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.285607 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 03:49:16.285666 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.285624 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 03:49:16.285666 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.285632 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 03:49:16.285666 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.285638 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 03:49:16.285666 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.285644 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 03:49:16.285666 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.285655 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 03:49:16.285666 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.285661 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 03:49:16.285666 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.285667 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 03:49:16.285870 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.285675 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 03:49:16.285870 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.285681 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 03:49:16.285870 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.285690 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 03:49:16.285870 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.285701 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 03:49:16.285870 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.285723 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 03:49:16.285870 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.285732 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 03:49:16.289538 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.289522 4966 server.go:1263] "Started kubelet" Nov 04 03:49:16.289676 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.289610 4966 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 03:49:16.289732 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.289704 4966 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 03:49:16.290679 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.289815 4966 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 03:49:16.290679 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.290307 4966 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 03:49:16.289999 ip-10-0-138-9 systemd[1]: Started Kubernetes Kubelet. Nov 04 03:49:16.291618 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.291597 4966 server.go:317] "Adding debug handlers to kubelet server" Nov 04 03:49:16.293776 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.293759 4966 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 03:49:16.293876 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.293855 4966 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 03:49:16.293953 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.293897 4966 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 03:49:16.294085 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.294069 4966 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 03:49:16.294564 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.294545 4966 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 03:49:16.294635 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.294596 4966 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 03:43:37 +0000 UTC" deadline="2028-04-22 09:48:42.097415916 +0000 UTC" Nov 04 03:49:16.294635 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.294625 4966 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12845h59m25.802793283s" Nov 04 03:49:16.295116 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:16.294789 4966 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-9.us-east-2.compute.internal\" not found" Nov 04 03:49:16.298014 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.295977 4966 factory.go:55] Registering systemd factory Nov 04 03:49:16.298014 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.295996 4966 factory.go:223] Registration of the systemd container factory successfully Nov 04 03:49:16.298014 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.296250 4966 factory.go:153] Registering CRI-O factory Nov 04 03:49:16.298014 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.296264 4966 factory.go:223] Registration of the crio container factory successfully Nov 04 03:49:16.298014 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.296338 4966 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 03:49:16.298014 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.296358 4966 factory.go:103] Registering Raw factory Nov 04 03:49:16.298014 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.296372 4966 manager.go:1199] Started watching for new ooms in manager Nov 04 03:49:16.298014 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.296739 4966 manager.go:319] Starting recovery of all containers Nov 04 03:49:16.298710 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.298681 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ab83f7b-3396-4cc7-875e-c2cc62696ff5" volumeName="kubernetes.io/projected/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-kube-api-access-4vf5x" seLinuxMountContext="" Nov 04 03:49:16.298772 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.298711 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c2ac1f5-6e08-48e3-ac83-a55f9181f4be" volumeName="kubernetes.io/projected/1c2ac1f5-6e08-48e3-ac83-a55f9181f4be-kube-api-access-zhvtl" seLinuxMountContext="" Nov 04 03:49:16.298772 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.298727 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26bb99fd-d223-4642-b147-b22bebd00835" volumeName="kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.298747 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="341692ff-c7ac-4387-b9a3-d41ddd8f2ad3" volumeName="kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-service-ca-bundle" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.298996 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="341692ff-c7ac-4387-b9a3-d41ddd8f2ad3" volumeName="kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-trusted-ca-bundle" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.299022 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b456f4f-071a-4e5f-b8b6-51c3ee790656" volumeName="kubernetes.io/projected/3b456f4f-071a-4e5f-b8b6-51c3ee790656-kube-api-access-s6hlp" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.299049 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6" volumeName="kubernetes.io/secret/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-serving-cert" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.299066 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1ced90a-6b3d-4553-b05f-d4ec50e6af9b" volumeName="kubernetes.io/projected/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-kube-api-access-6dcsj" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.299096 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2052e7f-5ee8-4066-bb29-cd227fa0965b" volumeName="kubernetes.io/secret/d2052e7f-5ee8-4066-bb29-cd227fa0965b-serving-cert" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.299117 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19970c28-9dee-4377-8e98-1fc65473d3e1" volumeName="kubernetes.io/configmap/19970c28-9dee-4377-8e98-1fc65473d3e1-ovnkube-config" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.299223 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b456f4f-071a-4e5f-b8b6-51c3ee790656" volumeName="kubernetes.io/secret/3b456f4f-071a-4e5f-b8b6-51c3ee790656-serving-cert" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.299251 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d313ca1-e640-42d9-a9f7-5e3e8918dc77" volumeName="kubernetes.io/projected/3d313ca1-e640-42d9-a9f7-5e3e8918dc77-kube-api-access-rw88q" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.299266 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c12382e2-90c4-45bf-bdd8-27653073cc5d" volumeName="kubernetes.io/empty-dir/c12382e2-90c4-45bf-bdd8-27653073cc5d-tmp-dir" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.299289 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d25872c4-1eee-48b8-a4be-9384dae267ec" volumeName="kubernetes.io/projected/d25872c4-1eee-48b8-a4be-9384dae267ec-kube-api-access-dfk6m" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.299312 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="341692ff-c7ac-4387-b9a3-d41ddd8f2ad3" volumeName="kubernetes.io/projected/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-kube-api-access-lsvdh" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.299335 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b456f4f-071a-4e5f-b8b6-51c3ee790656" volumeName="kubernetes.io/configmap/3b456f4f-071a-4e5f-b8b6-51c3ee790656-service-ca-bundle" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.299347 4966 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.299352 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b632ca9-a695-450e-a1a9-b5357ebf4d7d" volumeName="kubernetes.io/configmap/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-config-volume" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.299383 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcc0d689-48b6-467e-bdd0-6a03878a52b6" volumeName="kubernetes.io/secret/bcc0d689-48b6-467e-bdd0-6a03878a52b6-cert" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.299906 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c12382e2-90c4-45bf-bdd8-27653073cc5d" volumeName="kubernetes.io/projected/c12382e2-90c4-45bf-bdd8-27653073cc5d-kube-api-access-lj5tf" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.299953 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="366aeead-59de-4180-8d14-e8b821d04e22" volumeName="kubernetes.io/secret/366aeead-59de-4180-8d14-e8b821d04e22-signing-key" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.299986 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ce83f95-758c-4fd2-9678-441fe4309da7" volumeName="kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-kube-api-access-sqp55" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300004 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7a99c19-45c9-4294-b2af-42d20306ff28" volumeName="kubernetes.io/projected/f7a99c19-45c9-4294-b2af-42d20306ff28-kube-api-access-w7b4p" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300027 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78e55dc9-354d-455f-9d02-b3cfd5885d65" volumeName="kubernetes.io/empty-dir/78e55dc9-354d-455f-9d02-b3cfd5885d65-data-volume" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300043 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="363d8742-c0a7-405c-9683-7b12afbd8bcc" volumeName="kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-certificates" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300059 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="363d8742-c0a7-405c-9683-7b12afbd8bcc" volumeName="kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-installation-pull-secrets" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300082 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b456f4f-071a-4e5f-b8b6-51c3ee790656" volumeName="kubernetes.io/empty-dir/3b456f4f-071a-4e5f-b8b6-51c3ee790656-snapshots" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300097 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d313ca1-e640-42d9-a9f7-5e3e8918dc77" volumeName="kubernetes.io/secret/3d313ca1-e640-42d9-a9f7-5e3e8918dc77-serving-cert" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300121 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f37b227d-fc9c-4a81-85b1-62c04243ad83" volumeName="kubernetes.io/configmap/f37b227d-fc9c-4a81-85b1-62c04243ad83-nginx-conf" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300188 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27d72728-5623-41e0-bee5-a09937f9f87a" volumeName="kubernetes.io/projected/27d72728-5623-41e0-bee5-a09937f9f87a-kube-api-access-kr88g" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300204 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30f18911-ab5a-44e3-88b2-6508325ceb39" volumeName="kubernetes.io/projected/30f18911-ab5a-44e3-88b2-6508325ceb39-kube-api-access-q5t7v" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300220 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="363d8742-c0a7-405c-9683-7b12afbd8bcc" volumeName="kubernetes.io/empty-dir/363d8742-c0a7-405c-9683-7b12afbd8bcc-ca-trust-extracted" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300245 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="363d8742-c0a7-405c-9683-7b12afbd8bcc" volumeName="kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-kube-api-access-952rw" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300259 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6" volumeName="kubernetes.io/configmap/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-trusted-ca" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300280 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6" volumeName="kubernetes.io/projected/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-kube-api-access-zvg2z" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300296 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1ced90a-6b3d-4553-b05f-d4ec50e6af9b" volumeName="kubernetes.io/configmap/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300310 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7a99c19-45c9-4294-b2af-42d20306ff28" volumeName="kubernetes.io/configmap/f7a99c19-45c9-4294-b2af-42d20306ff28-cni-binary-copy" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300334 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19970c28-9dee-4377-8e98-1fc65473d3e1" volumeName="kubernetes.io/configmap/19970c28-9dee-4377-8e98-1fc65473d3e1-ovnkube-script-lib" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300348 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21ac7748-5e3d-427e-bd15-93cf4754cb0e" volumeName="kubernetes.io/projected/21ac7748-5e3d-427e-bd15-93cf4754cb0e-kube-api-access-5rpkk" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300367 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7714ee23-de1d-4e5d-88ba-cce91e08ba23" volumeName="kubernetes.io/projected/7714ee23-de1d-4e5d-88ba-cce91e08ba23-kube-api-access-fb5w2" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300383 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b632ca9-a695-450e-a1a9-b5357ebf4d7d" volumeName="kubernetes.io/projected/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-kube-api-access-szjzl" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300402 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93149be5-0d1f-4b08-b33e-d2e43bac726c" volumeName="kubernetes.io/secret/93149be5-0d1f-4b08-b33e-d2e43bac726c-agent-certs" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300423 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcc0d689-48b6-467e-bdd0-6a03878a52b6" volumeName="kubernetes.io/projected/bcc0d689-48b6-467e-bdd0-6a03878a52b6-kube-api-access-hjrzz" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300438 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b81b9bc-4eb5-48a7-98e1-6c7a98117f57" volumeName="kubernetes.io/projected/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-kube-api-access-knwrb" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300455 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d275592-23f8-4edc-aff7-016930318629" volumeName="kubernetes.io/empty-dir/2d275592-23f8-4edc-aff7-016930318629-etc-tuned" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300471 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="366aeead-59de-4180-8d14-e8b821d04e22" volumeName="kubernetes.io/configmap/366aeead-59de-4180-8d14-e8b821d04e22-signing-cabundle" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300483 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="795d3139-8d19-4852-bd9c-3c60b01a97f6" volumeName="kubernetes.io/projected/795d3139-8d19-4852-bd9c-3c60b01a97f6-kube-api-access-wnklb" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300506 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b632ca9-a695-450e-a1a9-b5357ebf4d7d" volumeName="kubernetes.io/empty-dir/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-tmp-dir" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300526 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19970c28-9dee-4377-8e98-1fc65473d3e1" volumeName="kubernetes.io/configmap/19970c28-9dee-4377-8e98-1fc65473d3e1-env-overrides" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300549 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ce83f95-758c-4fd2-9678-441fe4309da7" volumeName="kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-installation-pull-secrets" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300566 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78e55dc9-354d-455f-9d02-b3cfd5885d65" volumeName="kubernetes.io/projected/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-api-access-6w8m8" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300581 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2052e7f-5ee8-4066-bb29-cd227fa0965b" volumeName="kubernetes.io/configmap/d2052e7f-5ee8-4066-bb29-cd227fa0965b-config" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300603 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0df87f5d-31f4-4571-82b6-fe75c434ff27" volumeName="kubernetes.io/configmap/0df87f5d-31f4-4571-82b6-fe75c434ff27-telemetry-config" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300620 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f0a14d4-9771-4aa2-a6bd-87f5f73ee931" volumeName="kubernetes.io/projected/1f0a14d4-9771-4aa2-a6bd-87f5f73ee931-kube-api-access-sf4rk" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300639 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="341692ff-c7ac-4387-b9a3-d41ddd8f2ad3" volumeName="kubernetes.io/empty-dir/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-archives-path" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300659 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ce83f95-758c-4fd2-9678-441fe4309da7" volumeName="kubernetes.io/empty-dir/3ce83f95-758c-4fd2-9678-441fe4309da7-ca-trust-extracted" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300676 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1ced90a-6b3d-4553-b05f-d4ec50e6af9b" volumeName="kubernetes.io/configmap/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-cni-binary-copy" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300716 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2052e7f-5ee8-4066-bb29-cd227fa0965b" volumeName="kubernetes.io/projected/d2052e7f-5ee8-4066-bb29-cd227fa0965b-kube-api-access-pcjkb" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300729 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d25872c4-1eee-48b8-a4be-9384dae267ec" volumeName="kubernetes.io/configmap/d25872c4-1eee-48b8-a4be-9384dae267ec-iptables-alerter-script" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300749 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7a99c19-45c9-4294-b2af-42d20306ff28" volumeName="kubernetes.io/configmap/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-daemon-config" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300765 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19970c28-9dee-4377-8e98-1fc65473d3e1" volumeName="kubernetes.io/secret/19970c28-9dee-4377-8e98-1fc65473d3e1-ovn-node-metrics-cert" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300778 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ce83f95-758c-4fd2-9678-441fe4309da7" volumeName="kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-bound-sa-token" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300796 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d275592-23f8-4edc-aff7-016930318629" volumeName="kubernetes.io/projected/2d275592-23f8-4edc-aff7-016930318629-kube-api-access-kgv6q" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300811 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30f18911-ab5a-44e3-88b2-6508325ceb39" volumeName="kubernetes.io/configmap/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-config" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300834 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="363d8742-c0a7-405c-9683-7b12afbd8bcc" volumeName="kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-bound-sa-token" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300850 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391e5382-7119-4085-9939-cd81fc8b2d73" volumeName="kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300865 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b456f4f-071a-4e5f-b8b6-51c3ee790656" volumeName="kubernetes.io/configmap/3b456f4f-071a-4e5f-b8b6-51c3ee790656-trusted-ca-bundle" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300884 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6" volumeName="kubernetes.io/configmap/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-config" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300896 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1ced90a-6b3d-4553-b05f-d4ec50e6af9b" volumeName="kubernetes.io/configmap/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-cni-sysctl-allowlist" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300914 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fbef9e3-d5de-4a35-8745-2b1dc357c52b" volumeName="kubernetes.io/projected/3fbef9e3-d5de-4a35-8745-2b1dc357c52b-kube-api-access-rjgp4" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300944 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19970c28-9dee-4377-8e98-1fc65473d3e1" volumeName="kubernetes.io/projected/19970c28-9dee-4377-8e98-1fc65473d3e1-kube-api-access-n8q89" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300958 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b81b9bc-4eb5-48a7-98e1-6c7a98117f57" volumeName="kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-stats-auth" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300980 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b456f4f-071a-4e5f-b8b6-51c3ee790656" volumeName="kubernetes.io/empty-dir/3b456f4f-071a-4e5f-b8b6-51c3ee790656-tmp" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.300996 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0df87f5d-31f4-4571-82b6-fe75c434ff27" volumeName="kubernetes.io/projected/0df87f5d-31f4-4571-82b6-fe75c434ff27-kube-api-access-w9fvw" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.301020 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d275592-23f8-4edc-aff7-016930318629" volumeName="kubernetes.io/empty-dir/2d275592-23f8-4edc-aff7-016930318629-tmp" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.301033 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="366aeead-59de-4180-8d14-e8b821d04e22" volumeName="kubernetes.io/projected/366aeead-59de-4180-8d14-e8b821d04e22-kube-api-access-8852w" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.301047 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ce83f95-758c-4fd2-9678-441fe4309da7" volumeName="kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-certificates" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.301069 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d313ca1-e640-42d9-a9f7-5e3e8918dc77" volumeName="kubernetes.io/configmap/3d313ca1-e640-42d9-a9f7-5e3e8918dc77-config" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.301083 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b81b9bc-4eb5-48a7-98e1-6c7a98117f57" volumeName="kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-default-certificate" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.301105 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="363d8742-c0a7-405c-9683-7b12afbd8bcc" volumeName="kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-trusted-ca" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.301119 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="363d8742-c0a7-405c-9683-7b12afbd8bcc" volumeName="kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-image-registry-private-configuration" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.301144 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ce83f95-758c-4fd2-9678-441fe4309da7" volumeName="kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-trusted-ca" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.301165 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ce83f95-758c-4fd2-9678-441fe4309da7" volumeName="kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-image-registry-private-configuration" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.301179 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="795d3139-8d19-4852-bd9c-3c60b01a97f6" volumeName="kubernetes.io/configmap/795d3139-8d19-4852-bd9c-3c60b01a97f6-serviceca" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.301198 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93149be5-0d1f-4b08-b33e-d2e43bac726c" volumeName="kubernetes.io/configmap/93149be5-0d1f-4b08-b33e-d2e43bac726c-konnectivity-ca" seLinuxMountContext="" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.301214 4966 reconstruct.go:97] "Volume reconstruction finished" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.301223 4966 reconciler.go:29] "Reconciler: start to sync state" Nov 04 03:49:16.302094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.301755 4966 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:49:16.307467 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.304578 4966 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:49:16.307467 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.304886 4966 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 04 03:49:16.320383 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.320353 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/1.log" Nov 04 03:49:16.320383 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.320384 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/1.log" Nov 04 03:49:16.331739 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.331679 4966 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 03:49:16.334686 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.333253 4966 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 03:49:16.334686 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.333275 4966 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 03:49:16.334686 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.333295 4966 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 03:49:16.334686 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:16.333338 4966 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 04 03:49:16.337464 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.337413 4966 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:49:16.344576 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.344555 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/1.log" Nov 04 03:49:16.344674 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.344585 4966 generic.go:356] "Generic (PLEG): container finished" podID="9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6" containerID="a269cc6bda79ea6e7251d9cfa350a30df870c64031267278c93cb9a0f5bd17e5" exitCode=255 Nov 04 03:49:16.370091 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.370063 4966 generic.go:356] "Generic (PLEG): container finished" podID="a1ced90a-6b3d-4553-b05f-d4ec50e6af9b" containerID="93f5523a3a2e128d54a1f428fa567b19341efd7fd1b097e2ef6ed7b749966f2a" exitCode=0 Nov 04 03:49:16.370091 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.370088 4966 generic.go:356] "Generic (PLEG): container finished" podID="a1ced90a-6b3d-4553-b05f-d4ec50e6af9b" containerID="386748536b10856fc1342a279fd31808cbe188b6b7d4cb276c8ba739dba32fc5" exitCode=0 Nov 04 03:49:16.370091 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.370094 4966 generic.go:356] "Generic (PLEG): container finished" podID="a1ced90a-6b3d-4553-b05f-d4ec50e6af9b" containerID="3d7f6f1b17c3aeb902e2a9fc40375fd5d4149a3332c6d70ff29b9c9e5f2aa994" exitCode=0 Nov 04 03:49:16.370280 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.370100 4966 generic.go:356] "Generic (PLEG): container finished" podID="a1ced90a-6b3d-4553-b05f-d4ec50e6af9b" containerID="5c94be59ea87683b74e08b1e0652c8cd15bb80c8e0da0857e0111e5f4ac699dd" exitCode=0 Nov 04 03:49:16.370280 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.370106 4966 generic.go:356] "Generic (PLEG): container finished" podID="a1ced90a-6b3d-4553-b05f-d4ec50e6af9b" containerID="f61e71aafb6a895d7bf75d0a378ab632014e5e925ea34dfed6a0b188a5f2ffd6" exitCode=0 Nov 04 03:49:16.370280 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.370111 4966 generic.go:356] "Generic (PLEG): container finished" podID="a1ced90a-6b3d-4553-b05f-d4ec50e6af9b" containerID="6b3b988aa3691a5eeeb37794b1c0958a74ba2d97398c7b483b2433cffca61ab5" exitCode=0 Nov 04 03:49:16.386757 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.386730 4966 generic.go:356] "Generic (PLEG): container finished" podID="c1d9b8a1217c8d6e24202560e65c65d7" containerID="0aac18d105a9e8bde98e426e49bb32667d1d4fe81a7505a750d6258dbc1e8ec3" exitCode=0 Nov 04 03:49:16.394736 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.394707 4966 manager.go:324] Recovery completed Nov 04 03:49:16.416376 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.416313 4966 cpu_manager.go:226] "Starting" policy="none" Nov 04 03:49:16.416376 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.416334 4966 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 03:49:16.416376 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.416354 4966 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 03:49:16.416538 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.416471 4966 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 04 03:49:16.416538 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.416482 4966 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 04 03:49:16.416538 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.416498 4966 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 04 03:49:16.416538 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.416503 4966 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 04 03:49:16.416538 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.416508 4966 policy_none.go:50] "Start" Nov 04 03:49:16.416538 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.416516 4966 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 03:49:16.416538 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.416525 4966 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 03:49:16.416944 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.416931 4966 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 04 03:49:16.416982 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.416944 4966 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 04 03:49:16.416982 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.416950 4966 policy_none.go:44] "Start" Nov 04 03:49:16.421413 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.421384 4966 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 03:49:16.423946 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.423930 4966 manager.go:341] "Starting Device Plugin manager" Nov 04 03:49:16.424410 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:16.423958 4966 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 03:49:16.424410 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.424408 4966 server.go:87] "Starting device plugin registration server" Nov 04 03:49:16.424672 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.424658 4966 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 03:49:16.424735 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.424669 4966 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 03:49:16.424799 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.424776 4966 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 03:49:16.424914 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.424875 4966 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 03:49:16.424997 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.424936 4966 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 03:49:16.424997 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.424947 4966 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 03:49:16.425086 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.424980 4966 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T03:49:16.424913118Z","UUID":"a6efdfb2-2956-4a01-a1ed-930431045dbd","Handler":null,"Name":"","Endpoint":""} Nov 04 03:49:16.426402 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.426383 4966 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 03:49:16.426402 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.426414 4966 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 03:49:16.426555 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:16.426521 4966 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 03:49:16.433454 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.433414 4966 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal"] Nov 04 03:49:16.437751 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.437730 4966 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:49:16.437872 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.437833 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal" event={"ID":"37008f3824659b029a90a4ae33f9a950","Type":"ContainerStarted","Data":"0991dbab3a77a90594ed04c9b3281fe729c4d76ca82aff5cd740d5937bc7219e"} Nov 04 03:49:16.438054 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.438031 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" event={"ID":"c1d9b8a1217c8d6e24202560e65c65d7","Type":"ContainerStarted","Data":"74c2662d4fa71d149b9f11e1b1828c2fcb7f031b738ce068de310cb729f3c63a"} Nov 04 03:49:16.438138 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.438060 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" event={"ID":"c1d9b8a1217c8d6e24202560e65c65d7","Type":"ContainerDied","Data":"0aac18d105a9e8bde98e426e49bb32667d1d4fe81a7505a750d6258dbc1e8ec3"} Nov 04 03:49:16.438138 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.438080 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" event={"ID":"c1d9b8a1217c8d6e24202560e65c65d7","Type":"ContainerStarted","Data":"02ac35b4cb7404a4fb3d5ae493b982826d38cb4f4ee19f82583104436f253f62"} Nov 04 03:49:16.438138 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.438132 4966 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:49:16.452067 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.452042 4966 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 03:49:16.452166 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:16.452101 4966 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:49:16.452741 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.452725 4966 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 03:49:16.452854 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:16.452766 4966 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:49:16.506475 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.506446 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1d9b8a1217c8d6e24202560e65c65d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal\" (UID: \"c1d9b8a1217c8d6e24202560e65c65d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:49:16.506475 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.506476 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1d9b8a1217c8d6e24202560e65c65d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal\" (UID: \"c1d9b8a1217c8d6e24202560e65c65d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:49:16.506610 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.506494 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37008f3824659b029a90a4ae33f9a950-config\") pod \"kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal\" (UID: \"37008f3824659b029a90a4ae33f9a950\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:49:16.525255 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.525233 4966 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:49:16.526503 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.526485 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:49:16.526563 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.526518 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:49:16.526563 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.526529 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 03:49:16.526628 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.526573 4966 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:49:16.539234 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.539212 4966 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:49:16.539306 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.539273 4966 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:49:16.606890 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.606857 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1d9b8a1217c8d6e24202560e65c65d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal\" (UID: \"c1d9b8a1217c8d6e24202560e65c65d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:49:16.606890 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.606889 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1d9b8a1217c8d6e24202560e65c65d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal\" (UID: \"c1d9b8a1217c8d6e24202560e65c65d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:49:16.607087 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.606908 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37008f3824659b029a90a4ae33f9a950-config\") pod \"kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal\" (UID: \"37008f3824659b029a90a4ae33f9a950\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:49:16.607087 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.606992 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1d9b8a1217c8d6e24202560e65c65d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal\" (UID: \"c1d9b8a1217c8d6e24202560e65c65d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:49:16.607087 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.607030 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37008f3824659b029a90a4ae33f9a950-config\") pod \"kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal\" (UID: \"37008f3824659b029a90a4ae33f9a950\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:49:16.607087 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:16.606994 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1d9b8a1217c8d6e24202560e65c65d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal\" (UID: \"c1d9b8a1217c8d6e24202560e65c65d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:49:17.280632 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.280608 4966 apiserver.go:52] "Watching apiserver" Nov 04 03:49:17.296454 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.296429 4966 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 03:49:17.297108 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.297080 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8","openshift-insights/periodic-gathering-xwdp2-kwf4s","openshift-image-registry/image-registry-6448ff747c-dlpnw","openshift-insights/insights-operator-7c4986d449-gm5s7","openshift-multus/multus-fsbpl","openshift-image-registry/image-registry-8648b894c6-t5rf2","openshift-insights/insights-runtime-extractor-zfjhp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl","kube-system/konnectivity-agent-jvqw2","openshift-service-ca/service-ca-5fffc97bdf-tr6w8","kube-system/global-pull-secret-syncer-59fsx","openshift-ingress/router-default-78ccf8f879-s4sx4","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm","openshift-network-diagnostics/network-check-source-6f78d59f99-5xh74","openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r","openshift-cluster-node-tuning-operator/tuned-hl8q9","kube-system/kas-connection-checker-685895c7c9-qq74s","openshift-network-diagnostics/network-check-target-pbsd4","openshift-multus/network-metrics-daemon-mwzdx","openshift-network-operator/iptables-alerter-28k9n","kube-system/kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal","openshift-network-console/networking-console-plugin-86c478bb88-2hhp2","openshift-image-registry/node-ca-jkpjj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-vxkvk","kube-system/kas-connection-checker-685895c7c9-8v2sd","openshift-dns/dns-default-pvrd4","openshift-multus/multus-additional-cni-plugins-2pkqx","openshift-console-operator/console-operator-7cd56b8dbc-rr5bv","kube-system/kas-connection-checker-685895c7c9-wc5zp","openshift-dns/node-resolver-c8xnp","openshift-ingress-canary/ingress-canary-chk4t","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vjhfk"] Nov 04 03:49:17.299240 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.299218 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:17.300327 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.300300 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:17.300773 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.300751 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:17.301256 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.301241 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:17.304652 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.303383 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 04 03:49:17.304652 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.303658 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 04 03:49:17.304652 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.303779 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 04 03:49:17.304652 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.304380 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 03:49:17.304652 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.304478 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 04 03:49:17.304984 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.304674 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2hhp2" Nov 04 03:49:17.305686 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.305664 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 03:49:17.305686 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.305679 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:49:17.307052 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.307034 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 03:49:17.311819 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.311781 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 04 03:49:17.313276 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.313253 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 04 03:49:17.313556 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.313538 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 04 03:49:17.313556 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.313554 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:49:17.313683 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.313575 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:49:17.313717 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.313706 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:49:17.315218 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.315197 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 04 03:49:17.315434 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.315410 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 04 03:49:17.316467 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.316449 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:17.316968 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.316954 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:17.318104 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.318087 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:17.319145 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.319126 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 03:49:17.319145 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.319139 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 04 03:49:17.319278 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.319158 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 03:49:17.319278 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.319175 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:49:17.319278 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.319195 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 04 03:49:17.319278 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.319200 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 04 03:49:17.319278 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.319210 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 04 03:49:17.319278 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.319224 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:49:17.319278 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.319217 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 03:49:17.319278 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.319241 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:49:17.319278 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.319260 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 04 03:49:17.319564 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.319286 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 03:49:17.319564 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.319363 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 04 03:49:17.319692 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.319673 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 04 03:49:17.320660 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.320638 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 04 03:49:17.320740 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.320642 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 04 03:49:17.320819 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.320782 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 03:49:17.320819 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.320800 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 03:49:17.321036 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.321015 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 03:49:17.321198 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.321179 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 04 03:49:17.321426 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.321410 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 04 03:49:17.321630 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.321613 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 04 03:49:17.321794 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.321779 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 03:49:17.321999 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.321984 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 04 03:49:17.322106 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.322046 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 03:49:17.322402 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.322380 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 04 03:49:17.322512 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.322496 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 03:49:17.322635 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.322606 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 04 03:49:17.322688 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.322649 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 04 03:49:17.322731 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.322715 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-x27cl\"" Nov 04 03:49:17.322817 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.322799 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lvdkb\"" Nov 04 03:49:17.322883 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.322865 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 04 03:49:17.322943 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.322889 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9xsbw\"" Nov 04 03:49:17.323069 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.323054 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-75zvg\"" Nov 04 03:49:17.323123 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.323116 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 03:49:17.323190 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.323176 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 03:49:17.323250 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.323237 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-k94c2\"" Nov 04 03:49:17.323301 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.323275 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 04 03:49:17.323424 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.323408 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-slzqd\"" Nov 04 03:49:17.323496 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.323479 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-b8lr8\"" Nov 04 03:49:17.323697 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.323677 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 04 03:49:17.323801 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.323783 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 03:49:17.323955 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.323933 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 03:49:17.324063 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.324048 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:49:17.324138 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.323956 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 03:49:17.324225 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.324209 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-75rnz\"" Nov 04 03:49:17.324274 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.324239 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kk77b\"" Nov 04 03:49:17.324369 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.324352 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 03:49:17.324536 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.324522 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 03:49:17.324669 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.324654 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 03:49:17.324749 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.324729 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 03:49:17.324799 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.324762 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 03:49:17.324844 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.324711 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 04 03:49:17.324887 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.324764 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 03:49:17.324887 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.324733 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 03:49:17.325017 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.324899 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 03:49:17.325017 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.324873 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 03:49:17.325110 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.325055 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 03:49:17.325180 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.324685 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 03:49:17.325574 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.325302 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 03:49:17.325574 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.325320 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 03:49:17.325574 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.325532 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 03:49:17.325727 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.325656 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 03:49:17.325818 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.325805 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 04 03:49:17.325850 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.325832 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 04 03:49:17.326236 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.326221 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 04 03:49:17.326271 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.326261 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 03:49:17.390258 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.390236 4966 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:49:17.396690 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.396671 4966 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 03:49:17.403669 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.403650 4966 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 03:49:17.403774 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:17.403722 4966 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-9.us-east-2.compute.internal" Nov 04 03:49:17.408252 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.408231 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 04 03:49:17.408334 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.408288 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tl6ng\"" Nov 04 03:49:17.408512 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.408498 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 03:49:17.408694 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.408681 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 03:49:17.409004 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.408982 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-p2j7s\"" Nov 04 03:49:17.409856 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.409839 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 03:49:17.409972 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.409868 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m5nwx\"" Nov 04 03:49:17.409972 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.409903 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-49g8t\"" Nov 04 03:49:17.409972 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.409943 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-q9vm5\"" Nov 04 03:49:17.409972 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.409967 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j5cz2\"" Nov 04 03:49:17.410134 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.409978 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-49k7r\"" Nov 04 03:49:17.410134 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.410028 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-twz4j\"" Nov 04 03:49:17.410353 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.410336 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n475p\"" Nov 04 03:49:17.410411 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.410361 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ccqtd\"" Nov 04 03:49:17.410411 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.410387 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 04 03:49:17.410411 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.410389 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 03:49:17.410411 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.410395 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-78gsl\"" Nov 04 03:49:17.410546 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.410451 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4ttlw\"" Nov 04 03:49:17.410638 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.410614 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fmfbg\"" Nov 04 03:49:17.410638 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.410626 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 03:49:17.410638 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.410633 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-qdtd9\"" Nov 04 03:49:17.410799 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.410640 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 03:49:17.410799 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.410675 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 04 03:49:17.410799 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.410677 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pb77x\"" Nov 04 03:49:17.410799 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.410697 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 03:49:17.410799 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.410698 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 03:49:17.410799 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.410709 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vzqjs\"" Nov 04 03:49:17.410799 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.410678 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x8rgd\"" Nov 04 03:49:17.410799 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.410737 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-c9pqc\"" Nov 04 03:49:17.411772 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.411754 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-lib-modules\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.411822 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.411783 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szjzl\" (UniqueName: \"kubernetes.io/projected/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-kube-api-access-szjzl\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:17.411822 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.411801 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcjkb\" (UniqueName: \"kubernetes.io/projected/d2052e7f-5ee8-4066-bb29-cd227fa0965b-kube-api-access-pcjkb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm\" (UID: \"d2052e7f-5ee8-4066-bb29-cd227fa0965b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" Nov 04 03:49:17.411952 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.411824 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-cni-dir\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.411952 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.411894 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-conf-dir\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.412040 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.411944 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-image-registry-private-configuration\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:17.412040 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.411976 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:49:17.412040 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412003 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-systemd\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.412176 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412051 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w8m8\" (UniqueName: \"kubernetes.io/projected/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-api-access-6w8m8\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:17.412176 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412077 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf4rk\" (UniqueName: \"kubernetes.io/projected/1f0a14d4-9771-4aa2-a6bd-87f5f73ee931-kube-api-access-sf4rk\") pod \"kas-connection-checker-685895c7c9-qq74s\" (UID: \"1f0a14d4-9771-4aa2-a6bd-87f5f73ee931\") " pod="kube-system/kas-connection-checker-685895c7c9-qq74s" Nov 04 03:49:17.412176 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412104 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0df87f5d-31f4-4571-82b6-fe75c434ff27-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qmh6r\" (UID: \"0df87f5d-31f4-4571-82b6-fe75c434ff27\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:17.412318 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412197 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-tmp-dir\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:17.412318 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412269 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-image-registry-private-configuration\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:17.412318 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412279 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7a99c19-45c9-4294-b2af-42d20306ff28-cni-binary-copy\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.412318 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412310 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:49:17.412474 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412339 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-installation-pull-secrets\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:17.412474 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412315 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-tmp-dir\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:17.412474 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412451 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-certificates\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:17.412597 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412486 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsvdh\" (UniqueName: \"kubernetes.io/projected/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-kube-api-access-lsvdh\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:17.412597 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412508 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-var-lib-kubelet\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.412597 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412532 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-bound-sa-token\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:17.412597 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412535 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-installation-pull-secrets\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:17.412597 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412550 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-serving-cert\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:17.412810 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412626 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7a99c19-45c9-4294-b2af-42d20306ff28-cni-binary-copy\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.412810 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412653 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c12382e2-90c4-45bf-bdd8-27653073cc5d-hosts-file\") pod \"node-resolver-c8xnp\" (UID: \"c12382e2-90c4-45bf-bdd8-27653073cc5d\") " pod="openshift-dns/node-resolver-c8xnp" Nov 04 03:49:17.412810 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412684 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7b4p\" (UniqueName: \"kubernetes.io/projected/f7a99c19-45c9-4294-b2af-42d20306ff28-kube-api-access-w7b4p\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.412810 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412691 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-certificates\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:17.412810 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412711 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d313ca1-e640-42d9-a9f7-5e3e8918dc77-serving-cert\") pod \"service-ca-operator-544d749c8c-2jgg8\" (UID: \"3d313ca1-e640-42d9-a9f7-5e3e8918dc77\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" Nov 04 03:49:17.412810 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412769 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-serving-cert\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:17.413132 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412883 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:17.413132 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412937 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-run-ovn\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.413132 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.412971 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.413132 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413012 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19970c28-9dee-4377-8e98-1fc65473d3e1-ovnkube-script-lib\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.413132 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413052 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/366aeead-59de-4180-8d14-e8b821d04e22-signing-key\") pod \"service-ca-5fffc97bdf-tr6w8\" (UID: \"366aeead-59de-4180-8d14-e8b821d04e22\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" Nov 04 03:49:17.413132 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413086 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d313ca1-e640-42d9-a9f7-5e3e8918dc77-serving-cert\") pod \"service-ca-operator-544d749c8c-2jgg8\" (UID: \"3d313ca1-e640-42d9-a9f7-5e3e8918dc77\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" Nov 04 03:49:17.413132 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413111 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:17.413363 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413130 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjgp4\" (UniqueName: \"kubernetes.io/projected/3fbef9e3-d5de-4a35-8745-2b1dc357c52b-kube-api-access-rjgp4\") pod \"volume-data-source-validator-58c9f5f46d-vjhfk\" (UID: \"3fbef9e3-d5de-4a35-8745-2b1dc357c52b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vjhfk" Nov 04 03:49:17.413363 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413186 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-run-netns\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.413363 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413295 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/366aeead-59de-4180-8d14-e8b821d04e22-signing-key\") pod \"service-ca-5fffc97bdf-tr6w8\" (UID: \"366aeead-59de-4180-8d14-e8b821d04e22\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" Nov 04 03:49:17.413363 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413337 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw88q\" (UniqueName: \"kubernetes.io/projected/3d313ca1-e640-42d9-a9f7-5e3e8918dc77-kube-api-access-rw88q\") pod \"service-ca-operator-544d749c8c-2jgg8\" (UID: \"3d313ca1-e640-42d9-a9f7-5e3e8918dc77\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" Nov 04 03:49:17.413363 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413359 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26bb99fd-d223-4642-b147-b22bebd00835-dbus\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:49:17.413542 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413376 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:49:17.413542 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413384 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19970c28-9dee-4377-8e98-1fc65473d3e1-ovnkube-script-lib\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.413542 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413399 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b456f4f-071a-4e5f-b8b6-51c3ee790656-serving-cert\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:17.413542 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413528 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs\") pod \"network-metrics-daemon-mwzdx\" (UID: \"9ab83f7b-3396-4cc7-875e-c2cc62696ff5\") " pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:49:17.413679 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413605 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f37b227d-fc9c-4a81-85b1-62c04243ad83-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2hhp2\" (UID: \"f37b227d-fc9c-4a81-85b1-62c04243ad83\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2hhp2" Nov 04 03:49:17.413720 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413689 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/93149be5-0d1f-4b08-b33e-d2e43bac726c-konnectivity-ca\") pod \"konnectivity-agent-jvqw2\" (UID: \"93149be5-0d1f-4b08-b33e-d2e43bac726c\") " pod="kube-system/konnectivity-agent-jvqw2" Nov 04 03:49:17.413720 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413711 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5t7v\" (UniqueName: \"kubernetes.io/projected/30f18911-ab5a-44e3-88b2-6508325ceb39-kube-api-access-q5t7v\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:17.413803 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413727 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19970c28-9dee-4377-8e98-1fc65473d3e1-ovn-node-metrics-cert\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.413803 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413742 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgv6q\" (UniqueName: \"kubernetes.io/projected/2d275592-23f8-4edc-aff7-016930318629-kube-api-access-kgv6q\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.413803 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413763 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcc0d689-48b6-467e-bdd0-6a03878a52b6-cert\") pod \"ingress-canary-chk4t\" (UID: \"bcc0d689-48b6-467e-bdd0-6a03878a52b6\") " pod="openshift-ingress-canary/ingress-canary-chk4t" Nov 04 03:49:17.413803 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413690 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26bb99fd-d223-4642-b147-b22bebd00835-original-pull-secret\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:49:17.414011 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413829 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-daemon-config\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.414011 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413883 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b456f4f-071a-4e5f-b8b6-51c3ee790656-serving-cert\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:17.414011 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.413898 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/93149be5-0d1f-4b08-b33e-d2e43bac726c-konnectivity-ca\") pod \"konnectivity-agent-jvqw2\" (UID: \"93149be5-0d1f-4b08-b33e-d2e43bac726c\") " pod="kube-system/konnectivity-agent-jvqw2" Nov 04 03:49:17.414170 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414065 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcc0d689-48b6-467e-bdd0-6a03878a52b6-cert\") pod \"ingress-canary-chk4t\" (UID: \"bcc0d689-48b6-467e-bdd0-6a03878a52b6\") " pod="openshift-ingress-canary/ingress-canary-chk4t" Nov 04 03:49:17.414170 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414095 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6hlp\" (UniqueName: \"kubernetes.io/projected/3b456f4f-071a-4e5f-b8b6-51c3ee790656-kube-api-access-s6hlp\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:17.414170 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414100 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-daemon-config\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.414170 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414120 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-service-ca-bundle\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:17.414170 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414136 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-stats-auth\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:17.414396 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414173 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-kubernetes\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.414396 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414193 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d25872c4-1eee-48b8-a4be-9384dae267ec-iptables-alerter-script\") pod \"iptables-alerter-28k9n\" (UID: \"d25872c4-1eee-48b8-a4be-9384dae267ec\") " pod="openshift-network-operator/iptables-alerter-28k9n" Nov 04 03:49:17.414396 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414215 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-run-systemd\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.414396 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414223 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19970c28-9dee-4377-8e98-1fc65473d3e1-ovn-node-metrics-cert\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.414396 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414231 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8q89\" (UniqueName: \"kubernetes.io/projected/19970c28-9dee-4377-8e98-1fc65473d3e1-kube-api-access-n8q89\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.414396 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414264 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjrzz\" (UniqueName: \"kubernetes.io/projected/bcc0d689-48b6-467e-bdd0-6a03878a52b6-kube-api-access-hjrzz\") pod \"ingress-canary-chk4t\" (UID: \"bcc0d689-48b6-467e-bdd0-6a03878a52b6\") " pod="openshift-ingress-canary/ingress-canary-chk4t" Nov 04 03:49:17.414396 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414277 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-service-ca-bundle\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:17.414396 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414299 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-var-lib-cni-multus\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.414396 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414341 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.414396 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414382 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-stats-auth\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:17.414769 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414410 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-trusted-ca\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:17.414769 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414467 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-system-cni-dir\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:49:17.414769 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414496 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12382e2-90c4-45bf-bdd8-27653073cc5d-tmp-dir\") pod \"node-resolver-c8xnp\" (UID: \"c12382e2-90c4-45bf-bdd8-27653073cc5d\") " pod="openshift-dns/node-resolver-c8xnp" Nov 04 03:49:17.414769 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414510 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d25872c4-1eee-48b8-a4be-9384dae267ec-iptables-alerter-script\") pod \"iptables-alerter-28k9n\" (UID: \"d25872c4-1eee-48b8-a4be-9384dae267ec\") " pod="openshift-network-operator/iptables-alerter-28k9n" Nov 04 03:49:17.414769 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414599 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-run-netns\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.414769 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414623 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12382e2-90c4-45bf-bdd8-27653073cc5d-tmp-dir\") pod \"node-resolver-c8xnp\" (UID: \"c12382e2-90c4-45bf-bdd8-27653073cc5d\") " pod="openshift-dns/node-resolver-c8xnp" Nov 04 03:49:17.414769 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414666 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b456f4f-071a-4e5f-b8b6-51c3ee790656-service-ca-bundle\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:17.414769 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414699 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-trusted-ca\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:17.414769 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414724 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-952rw\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-kube-api-access-952rw\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:17.414769 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414767 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-systemd-units\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.415189 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414791 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-etc-openvswitch\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.415189 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414833 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-trusted-ca\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:17.415189 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414842 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b456f4f-071a-4e5f-b8b6-51c3ee790656-service-ca-bundle\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:17.415189 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414832 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-cni-bin\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.415189 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414877 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:17.415189 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414942 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-bound-sa-token\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:17.415189 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.414983 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-metrics-certs\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:17.415189 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415031 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-sys\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.415189 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415058 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-socket-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.415189 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415086 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rpkk\" (UniqueName: \"kubernetes.io/projected/21ac7748-5e3d-427e-bd15-93cf4754cb0e-kube-api-access-5rpkk\") pod \"network-check-source-6f78d59f99-5xh74\" (UID: \"21ac7748-5e3d-427e-bd15-93cf4754cb0e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5xh74" Nov 04 03:49:17.415189 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415169 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d25872c4-1eee-48b8-a4be-9384dae267ec-host-slash\") pod \"iptables-alerter-28k9n\" (UID: \"d25872c4-1eee-48b8-a4be-9384dae267ec\") " pod="openshift-network-operator/iptables-alerter-28k9n" Nov 04 03:49:17.415189 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415165 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-trusted-ca\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:17.415503 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415271 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:17.415503 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415303 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnklb\" (UniqueName: \"kubernetes.io/projected/795d3139-8d19-4852-bd9c-3c60b01a97f6-kube-api-access-wnklb\") pod \"node-ca-jkpjj\" (UID: \"795d3139-8d19-4852-bd9c-3c60b01a97f6\") " pod="openshift-image-registry/node-ca-jkpjj" Nov 04 03:49:17.415503 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415326 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-var-lib-openvswitch\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.415503 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415399 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-trusted-ca\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:17.415503 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415429 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:17.415503 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415445 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-system-cni-dir\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.415658 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415598 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2d275592-23f8-4edc-aff7-016930318629-etc-tuned\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.415658 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415648 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78e55dc9-354d-455f-9d02-b3cfd5885d65-crio-socket\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:17.415754 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415684 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2d275592-23f8-4edc-aff7-016930318629-etc-tuned\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.415754 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415683 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9fvw\" (UniqueName: \"kubernetes.io/projected/0df87f5d-31f4-4571-82b6-fe75c434ff27-kube-api-access-w9fvw\") pod \"cluster-monitoring-operator-884576c57-qmh6r\" (UID: \"0df87f5d-31f4-4571-82b6-fe75c434ff27\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:17.415754 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415707 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2052e7f-5ee8-4066-bb29-cd227fa0965b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm\" (UID: \"d2052e7f-5ee8-4066-bb29-cd227fa0965b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" Nov 04 03:49:17.415754 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415742 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-trusted-ca\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:17.415944 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415813 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2052e7f-5ee8-4066-bb29-cd227fa0965b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm\" (UID: \"d2052e7f-5ee8-4066-bb29-cd227fa0965b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" Nov 04 03:49:17.415944 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415833 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vf5x\" (UniqueName: \"kubernetes.io/projected/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-kube-api-access-4vf5x\") pod \"network-metrics-daemon-mwzdx\" (UID: \"9ab83f7b-3396-4cc7-875e-c2cc62696ff5\") " pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:49:17.415944 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415873 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-kubelet\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.415944 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415902 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19970c28-9dee-4377-8e98-1fc65473d3e1-env-overrides\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.416114 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.415956 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvg2z\" (UniqueName: \"kubernetes.io/projected/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-kube-api-access-zvg2z\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:17.416114 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416044 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-image-registry-private-configuration\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:17.416114 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416079 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0df87f5d-31f4-4571-82b6-fe75c434ff27-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qmh6r\" (UID: \"0df87f5d-31f4-4571-82b6-fe75c434ff27\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:17.416114 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416091 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19970c28-9dee-4377-8e98-1fc65473d3e1-env-overrides\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.416300 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416110 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-run-multus-certs\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.416300 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416152 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b456f4f-071a-4e5f-b8b6-51c3ee790656-tmp\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:17.416300 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416179 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfk6m\" (UniqueName: \"kubernetes.io/projected/d25872c4-1eee-48b8-a4be-9384dae267ec-kube-api-access-dfk6m\") pod \"iptables-alerter-28k9n\" (UID: \"d25872c4-1eee-48b8-a4be-9384dae267ec\") " pod="openshift-network-operator/iptables-alerter-28k9n" Nov 04 03:49:17.416300 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416203 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-image-registry-private-configuration\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:17.416300 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416230 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f37b227d-fc9c-4a81-85b1-62c04243ad83-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2hhp2\" (UID: \"f37b227d-fc9c-4a81-85b1-62c04243ad83\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2hhp2" Nov 04 03:49:17.416300 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416248 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/363d8742-c0a7-405c-9683-7b12afbd8bcc-ca-trust-extracted\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:17.416300 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416253 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b456f4f-071a-4e5f-b8b6-51c3ee790656-tmp\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:17.416300 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416256 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0df87f5d-31f4-4571-82b6-fe75c434ff27-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qmh6r\" (UID: \"0df87f5d-31f4-4571-82b6-fe75c434ff27\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:17.416300 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416273 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dcsj\" (UniqueName: \"kubernetes.io/projected/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-kube-api-access-6dcsj\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:49:17.416300 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416302 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26bb99fd-d223-4642-b147-b22bebd00835-kubelet-config\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:49:17.416610 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416327 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:17.416610 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416330 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/363d8742-c0a7-405c-9683-7b12afbd8bcc-ca-trust-extracted\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:17.416610 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416417 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f37b227d-fc9c-4a81-85b1-62c04243ad83-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2hhp2\" (UID: \"f37b227d-fc9c-4a81-85b1-62c04243ad83\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2hhp2" Nov 04 03:49:17.416610 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416436 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:49:17.416610 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416467 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/366aeead-59de-4180-8d14-e8b821d04e22-signing-cabundle\") pod \"service-ca-5fffc97bdf-tr6w8\" (UID: \"366aeead-59de-4180-8d14-e8b821d04e22\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" Nov 04 03:49:17.416610 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416530 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d2052e7f-5ee8-4066-bb29-cd227fa0965b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm\" (UID: \"d2052e7f-5ee8-4066-bb29-cd227fa0965b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" Nov 04 03:49:17.416610 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416547 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/795d3139-8d19-4852-bd9c-3c60b01a97f6-host\") pod \"node-ca-jkpjj\" (UID: \"795d3139-8d19-4852-bd9c-3c60b01a97f6\") " pod="openshift-image-registry/node-ca-jkpjj" Nov 04 03:49:17.416610 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416561 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/795d3139-8d19-4852-bd9c-3c60b01a97f6-serviceca\") pod \"node-ca-jkpjj\" (UID: \"795d3139-8d19-4852-bd9c-3c60b01a97f6\") " pod="openshift-image-registry/node-ca-jkpjj" Nov 04 03:49:17.416610 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416576 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19970c28-9dee-4377-8e98-1fc65473d3e1-ovnkube-config\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.416610 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416595 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416622 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ce83f95-758c-4fd2-9678-441fe4309da7-ca-trust-extracted\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416635 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/366aeead-59de-4180-8d14-e8b821d04e22-signing-cabundle\") pod \"service-ca-5fffc97bdf-tr6w8\" (UID: \"366aeead-59de-4180-8d14-e8b821d04e22\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416650 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-node-log\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416676 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-cnibin\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416694 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ce83f95-758c-4fd2-9678-441fe4309da7-ca-trust-extracted\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416622 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416704 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416730 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3b456f4f-071a-4e5f-b8b6-51c3ee790656-snapshots\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416758 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-run-openvswitch\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416761 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d2052e7f-5ee8-4066-bb29-cd227fa0965b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm\" (UID: \"d2052e7f-5ee8-4066-bb29-cd227fa0965b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416784 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/795d3139-8d19-4852-bd9c-3c60b01a97f6-serviceca\") pod \"node-ca-jkpjj\" (UID: \"795d3139-8d19-4852-bd9c-3c60b01a97f6\") " pod="openshift-image-registry/node-ca-jkpjj" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416808 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3b456f4f-071a-4e5f-b8b6-51c3ee790656-snapshots\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416813 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19970c28-9dee-4377-8e98-1fc65473d3e1-ovnkube-config\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416842 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416890 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-config\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416907 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-installation-pull-secrets\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416941 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhvtl\" (UniqueName: \"kubernetes.io/projected/1c2ac1f5-6e08-48e3-ac83-a55f9181f4be-kube-api-access-zhvtl\") pod \"kas-connection-checker-685895c7c9-wc5zp\" (UID: \"1c2ac1f5-6e08-48e3-ac83-a55f9181f4be\") " pod="kube-system/kas-connection-checker-685895c7c9-wc5zp" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416958 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416973 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj5tf\" (UniqueName: \"kubernetes.io/projected/c12382e2-90c4-45bf-bdd8-27653073cc5d-kube-api-access-lj5tf\") pod \"node-resolver-c8xnp\" (UID: \"c12382e2-90c4-45bf-bdd8-27653073cc5d\") " pod="openshift-dns/node-resolver-c8xnp" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.416988 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-modprobe-d\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.417090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417003 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-sysctl-conf\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417103 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-config\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417117 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-installation-pull-secrets\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417144 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2fj\" (UniqueName: \"kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj\") pod \"network-check-target-pbsd4\" (UID: \"391e5382-7119-4085-9939-cd81fc8b2d73\") " pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417175 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr88g\" (UniqueName: \"kubernetes.io/projected/27d72728-5623-41e0-bee5-a09937f9f87a-kube-api-access-kr88g\") pod \"kas-connection-checker-685895c7c9-8v2sd\" (UID: \"27d72728-5623-41e0-bee5-a09937f9f87a\") " pod="kube-system/kas-connection-checker-685895c7c9-8v2sd" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417204 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knwrb\" (UniqueName: \"kubernetes.io/projected/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-kube-api-access-knwrb\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417232 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-sysctl-d\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417258 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d313ca1-e640-42d9-a9f7-5e3e8918dc77-config\") pod \"service-ca-operator-544d749c8c-2jgg8\" (UID: \"3d313ca1-e640-42d9-a9f7-5e3e8918dc77\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417284 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-log-socket\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417327 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-cni-netd\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417359 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-cni-binary-copy\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417438 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-config-volume\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417504 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-certificates\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417541 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-cni-binary-copy\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417559 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d313ca1-e640-42d9-a9f7-5e3e8918dc77-config\") pod \"service-ca-operator-544d749c8c-2jgg8\" (UID: \"3d313ca1-e640-42d9-a9f7-5e3e8918dc77\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417582 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/93149be5-0d1f-4b08-b33e-d2e43bac726c-agent-certs\") pod \"konnectivity-agent-jvqw2\" (UID: \"93149be5-0d1f-4b08-b33e-d2e43bac726c\") " pod="kube-system/konnectivity-agent-jvqw2" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417610 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d275592-23f8-4edc-aff7-016930318629-tmp\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417626 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-certificates\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417637 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-config-volume\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417636 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-device-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417697 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-trusted-ca-bundle\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417727 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-default-certificate\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417769 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-host\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.417804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417791 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d275592-23f8-4edc-aff7-016930318629-tmp\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417821 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/93149be5-0d1f-4b08-b33e-d2e43bac726c-agent-certs\") pod \"konnectivity-agent-jvqw2\" (UID: \"93149be5-0d1f-4b08-b33e-d2e43bac726c\") " pod="kube-system/konnectivity-agent-jvqw2" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417839 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78e55dc9-354d-455f-9d02-b3cfd5885d65-data-volume\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417885 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78e55dc9-354d-455f-9d02-b3cfd5885d65-data-volume\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417885 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-cnibin\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417937 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-default-certificate\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417938 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-run-k8s-cni-cncf-io\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417974 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-sys-fs\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.417996 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb5w2\" (UniqueName: \"kubernetes.io/projected/7714ee23-de1d-4e5d-88ba-cce91e08ba23-kube-api-access-fb5w2\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.418019 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-sysconfig\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.418043 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqp55\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-kube-api-access-sqp55\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.418060 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-registration-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.418068 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-trusted-ca-bundle\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.418075 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-var-lib-cni-bin\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.418141 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b456f4f-071a-4e5f-b8b6-51c3ee790656-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.418193 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-etc-selinux\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.418243 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-slash\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.418277 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-socket-dir-parent\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.418310 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-run\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.418341 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-var-lib-kubelet\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.418392 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8852w\" (UniqueName: \"kubernetes.io/projected/366aeead-59de-4180-8d14-e8b821d04e22-kube-api-access-8852w\") pod \"service-ca-5fffc97bdf-tr6w8\" (UID: \"366aeead-59de-4180-8d14-e8b821d04e22\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.418422 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-hostroot\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.418462 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-etc-kubernetes\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.418488 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.418505 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b456f4f-071a-4e5f-b8b6-51c3ee790656-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.418510 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-archives-path\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:17.418723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.418572 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-archives-path\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:17.438978 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.438956 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szjzl\" (UniqueName: \"kubernetes.io/projected/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-kube-api-access-szjzl\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:17.439491 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.439476 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7b4p\" (UniqueName: \"kubernetes.io/projected/f7a99c19-45c9-4294-b2af-42d20306ff28-kube-api-access-w7b4p\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.440080 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.440064 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcjkb\" (UniqueName: \"kubernetes.io/projected/d2052e7f-5ee8-4066-bb29-cd227fa0965b-kube-api-access-pcjkb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm\" (UID: \"d2052e7f-5ee8-4066-bb29-cd227fa0965b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" Nov 04 03:49:17.440549 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.440532 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsvdh\" (UniqueName: \"kubernetes.io/projected/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-kube-api-access-lsvdh\") pod \"periodic-gathering-xwdp2-kwf4s\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:49:17.441098 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.441082 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjgp4\" (UniqueName: \"kubernetes.io/projected/3fbef9e3-d5de-4a35-8745-2b1dc357c52b-kube-api-access-rjgp4\") pod \"volume-data-source-validator-58c9f5f46d-vjhfk\" (UID: \"3fbef9e3-d5de-4a35-8745-2b1dc357c52b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vjhfk" Nov 04 03:49:17.441218 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.441199 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-bound-sa-token\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:17.441468 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.441453 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf4rk\" (UniqueName: \"kubernetes.io/projected/1f0a14d4-9771-4aa2-a6bd-87f5f73ee931-kube-api-access-sf4rk\") pod \"kas-connection-checker-685895c7c9-qq74s\" (UID: \"1f0a14d4-9771-4aa2-a6bd-87f5f73ee931\") " pod="kube-system/kas-connection-checker-685895c7c9-qq74s" Nov 04 03:49:17.442357 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.442342 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w8m8\" (UniqueName: \"kubernetes.io/projected/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-api-access-6w8m8\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:17.450179 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.450150 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw88q\" (UniqueName: \"kubernetes.io/projected/3d313ca1-e640-42d9-a9f7-5e3e8918dc77-kube-api-access-rw88q\") pod \"service-ca-operator-544d749c8c-2jgg8\" (UID: \"3d313ca1-e640-42d9-a9f7-5e3e8918dc77\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" Nov 04 03:49:17.450257 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.450153 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr88g\" (UniqueName: \"kubernetes.io/projected/27d72728-5623-41e0-bee5-a09937f9f87a-kube-api-access-kr88g\") pod \"kas-connection-checker-685895c7c9-8v2sd\" (UID: \"27d72728-5623-41e0-bee5-a09937f9f87a\") " pod="kube-system/kas-connection-checker-685895c7c9-8v2sd" Nov 04 03:49:17.450471 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.450450 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqp55\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-kube-api-access-sqp55\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:17.450507 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.450456 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8852w\" (UniqueName: \"kubernetes.io/projected/366aeead-59de-4180-8d14-e8b821d04e22-kube-api-access-8852w\") pod \"service-ca-5fffc97bdf-tr6w8\" (UID: \"366aeead-59de-4180-8d14-e8b821d04e22\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tr6w8" Nov 04 03:49:17.455271 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.455250 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnklb\" (UniqueName: \"kubernetes.io/projected/795d3139-8d19-4852-bd9c-3c60b01a97f6-kube-api-access-wnklb\") pod \"node-ca-jkpjj\" (UID: \"795d3139-8d19-4852-bd9c-3c60b01a97f6\") " pod="openshift-image-registry/node-ca-jkpjj" Nov 04 03:49:17.455511 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.455487 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knwrb\" (UniqueName: \"kubernetes.io/projected/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-kube-api-access-knwrb\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:17.455571 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.455558 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-bound-sa-token\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:17.455655 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.455636 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vf5x\" (UniqueName: \"kubernetes.io/projected/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-kube-api-access-4vf5x\") pod \"network-metrics-daemon-mwzdx\" (UID: \"9ab83f7b-3396-4cc7-875e-c2cc62696ff5\") " pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:49:17.455686 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.455651 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6hlp\" (UniqueName: \"kubernetes.io/projected/3b456f4f-071a-4e5f-b8b6-51c3ee790656-kube-api-access-s6hlp\") pod \"insights-operator-7c4986d449-gm5s7\" (UID: \"3b456f4f-071a-4e5f-b8b6-51c3ee790656\") " pod="openshift-insights/insights-operator-7c4986d449-gm5s7" Nov 04 03:49:17.455834 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.455813 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhvtl\" (UniqueName: \"kubernetes.io/projected/1c2ac1f5-6e08-48e3-ac83-a55f9181f4be-kube-api-access-zhvtl\") pod \"kas-connection-checker-685895c7c9-wc5zp\" (UID: \"1c2ac1f5-6e08-48e3-ac83-a55f9181f4be\") " pod="kube-system/kas-connection-checker-685895c7c9-wc5zp" Nov 04 03:49:17.455865 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.455846 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfk6m\" (UniqueName: \"kubernetes.io/projected/d25872c4-1eee-48b8-a4be-9384dae267ec-kube-api-access-dfk6m\") pod \"iptables-alerter-28k9n\" (UID: \"d25872c4-1eee-48b8-a4be-9384dae267ec\") " pod="openshift-network-operator/iptables-alerter-28k9n" Nov 04 03:49:17.459315 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.459295 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgv6q\" (UniqueName: \"kubernetes.io/projected/2d275592-23f8-4edc-aff7-016930318629-kube-api-access-kgv6q\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.463403 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.460320 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rpkk\" (UniqueName: \"kubernetes.io/projected/21ac7748-5e3d-427e-bd15-93cf4754cb0e-kube-api-access-5rpkk\") pod \"network-check-source-6f78d59f99-5xh74\" (UID: \"21ac7748-5e3d-427e-bd15-93cf4754cb0e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5xh74" Nov 04 03:49:17.463403 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.460778 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb5w2\" (UniqueName: \"kubernetes.io/projected/7714ee23-de1d-4e5d-88ba-cce91e08ba23-kube-api-access-fb5w2\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.464280 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.464260 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj5tf\" (UniqueName: \"kubernetes.io/projected/c12382e2-90c4-45bf-bdd8-27653073cc5d-kube-api-access-lj5tf\") pod \"node-resolver-c8xnp\" (UID: \"c12382e2-90c4-45bf-bdd8-27653073cc5d\") " pod="openshift-dns/node-resolver-c8xnp" Nov 04 03:49:17.464387 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.464370 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjrzz\" (UniqueName: \"kubernetes.io/projected/bcc0d689-48b6-467e-bdd0-6a03878a52b6-kube-api-access-hjrzz\") pod \"ingress-canary-chk4t\" (UID: \"bcc0d689-48b6-467e-bdd0-6a03878a52b6\") " pod="openshift-ingress-canary/ingress-canary-chk4t" Nov 04 03:49:17.464547 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.464530 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvg2z\" (UniqueName: \"kubernetes.io/projected/9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6-kube-api-access-zvg2z\") pod \"console-operator-7cd56b8dbc-rr5bv\" (UID: \"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:17.466910 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.466890 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq2fj\" (UniqueName: \"kubernetes.io/projected/391e5382-7119-4085-9939-cd81fc8b2d73-kube-api-access-mq2fj\") pod \"network-check-target-pbsd4\" (UID: \"391e5382-7119-4085-9939-cd81fc8b2d73\") " pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:49:17.467152 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.467137 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-952rw\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-kube-api-access-952rw\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:17.467882 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.467865 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5t7v\" (UniqueName: \"kubernetes.io/projected/30f18911-ab5a-44e3-88b2-6508325ceb39-kube-api-access-q5t7v\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:17.468606 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.468589 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8q89\" (UniqueName: \"kubernetes.io/projected/19970c28-9dee-4377-8e98-1fc65473d3e1-kube-api-access-n8q89\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.468839 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.468822 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9fvw\" (UniqueName: \"kubernetes.io/projected/0df87f5d-31f4-4571-82b6-fe75c434ff27-kube-api-access-w9fvw\") pod \"cluster-monitoring-operator-884576c57-qmh6r\" (UID: \"0df87f5d-31f4-4571-82b6-fe75c434ff27\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:17.471542 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.471526 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dcsj\" (UniqueName: \"kubernetes.io/projected/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-kube-api-access-6dcsj\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:49:17.519301 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519278 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs\") pod \"network-metrics-daemon-mwzdx\" (UID: \"9ab83f7b-3396-4cc7-875e-c2cc62696ff5\") " pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:49:17.519406 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519308 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f37b227d-fc9c-4a81-85b1-62c04243ad83-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2hhp2\" (UID: \"f37b227d-fc9c-4a81-85b1-62c04243ad83\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2hhp2" Nov 04 03:49:17.519406 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519329 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-kubernetes\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.519406 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519345 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-run-systemd\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.519406 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519391 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-kubernetes\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.519564 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519553 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-run-systemd\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.519605 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519573 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-var-lib-cni-multus\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.519605 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519589 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.519695 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519606 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-system-cni-dir\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:49:17.519695 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519621 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-var-lib-cni-multus\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.519695 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519624 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-run-netns\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.519695 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519654 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-run-netns\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.519695 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519663 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-systemd-units\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.519695 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519664 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.519695 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519673 4966 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 03:49:17.519695 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519687 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-system-cni-dir\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:49:17.520055 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519707 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-etc-openvswitch\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.520055 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519680 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-etc-openvswitch\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.520055 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519714 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-systemd-units\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.520055 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519753 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-cni-bin\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.520055 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519781 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:17.520055 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519806 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-metrics-certs\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:17.520055 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519815 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-cni-bin\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.520055 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519830 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-sys\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.520055 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:17.519869 4966 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:49:17.520055 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:17.519886 4966 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6448ff747c-dlpnw: secret "image-registry-tls" not found Nov 04 03:49:17.520055 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519908 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-sys\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.520055 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519944 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-socket-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.520055 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.519873 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-socket-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.520055 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:17.519981 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls podName:363d8742-c0a7-405c-9683-7b12afbd8bcc nodeName:}" failed. No retries permitted until 2026-11-04 03:49:18.019967048 +0000 UTC m=+1.820912850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls") pod "image-registry-6448ff747c-dlpnw" (UID: "363d8742-c0a7-405c-9683-7b12afbd8bcc") : secret "image-registry-tls" not found Nov 04 03:49:17.520055 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520010 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d25872c4-1eee-48b8-a4be-9384dae267ec-host-slash\") pod \"iptables-alerter-28k9n\" (UID: \"d25872c4-1eee-48b8-a4be-9384dae267ec\") " pod="openshift-network-operator/iptables-alerter-28k9n" Nov 04 03:49:17.520055 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520035 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:17.520055 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520055 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d25872c4-1eee-48b8-a4be-9384dae267ec-host-slash\") pod \"iptables-alerter-28k9n\" (UID: \"d25872c4-1eee-48b8-a4be-9384dae267ec\") " pod="openshift-network-operator/iptables-alerter-28k9n" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520067 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-var-lib-openvswitch\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520093 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520130 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-system-cni-dir\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520150 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-var-lib-openvswitch\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:17.520180 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle podName:1b81b9bc-4eb5-48a7-98e1-6c7a98117f57 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:18.020165487 +0000 UTC m=+1.821111291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle") pod "router-default-78ccf8f879-s4sx4" (UID: "1b81b9bc-4eb5-48a7-98e1-6c7a98117f57") : configmap references non-existent config key: service-ca.crt Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520196 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78e55dc9-354d-455f-9d02-b3cfd5885d65-crio-socket\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520231 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-system-cni-dir\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:17.520253 4966 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520153 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78e55dc9-354d-455f-9d02-b3cfd5885d65-crio-socket\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:17.520322 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls podName:7b632ca9-a695-450e-a1a9-b5357ebf4d7d nodeName:}" failed. No retries permitted until 2026-11-04 03:49:18.020303337 +0000 UTC m=+1.821249148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls") pod "dns-default-pvrd4" (UID: "7b632ca9-a695-450e-a1a9-b5357ebf4d7d") : secret "dns-default-metrics-tls" not found Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520343 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-kubelet\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520376 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-run-multus-certs\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520403 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-kubelet\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520405 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26bb99fd-d223-4642-b147-b22bebd00835-kubelet-config\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520433 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-run-multus-certs\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520439 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520477 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/795d3139-8d19-4852-bd9c-3c60b01a97f6-host\") pod \"node-ca-jkpjj\" (UID: \"795d3139-8d19-4852-bd9c-3c60b01a97f6\") " pod="openshift-image-registry/node-ca-jkpjj" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520477 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26bb99fd-d223-4642-b147-b22bebd00835-kubelet-config\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:17.520502 4966 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520514 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/795d3139-8d19-4852-bd9c-3c60b01a97f6-host\") pod \"node-ca-jkpjj\" (UID: \"795d3139-8d19-4852-bd9c-3c60b01a97f6\") " pod="openshift-image-registry/node-ca-jkpjj" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520534 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:17.520543 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls podName:30f18911-ab5a-44e3-88b2-6508325ceb39 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:18.020530471 +0000 UTC m=+1.821476271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6x2fz" (UID: "30f18911-ab5a-44e3-88b2-6508325ceb39") : secret "samples-operator-tls" not found Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520580 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-node-log\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:17.520594 4966 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520606 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-cnibin\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:17.520621 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls podName:78e55dc9-354d-455f-9d02-b3cfd5885d65 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:18.020613151 +0000 UTC m=+1.821558994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zfjhp" (UID: "78e55dc9-354d-455f-9d02-b3cfd5885d65") : secret "insights-runtime-extractor-tls" not found Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520638 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520642 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-node-log\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520651 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-cnibin\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520681 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-run-openvswitch\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:17.520690 4966 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520706 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:17.520712 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-rbac-proxy-cm podName:78e55dc9-354d-455f-9d02-b3cfd5885d65 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:18.020705783 +0000 UTC m=+1.821651608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zfjhp" (UID: "78e55dc9-354d-455f-9d02-b3cfd5885d65") : configmap "kube-rbac-proxy" not found Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520741 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-run-openvswitch\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520744 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520756 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520776 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-modprobe-d\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520794 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-sysctl-conf\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:17.520812 4966 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:17.520824 4966 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8648b894c6-t5rf2: secret "image-registry-tls" not found Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520859 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-sysctl-d\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:17.520871 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls podName:3ce83f95-758c-4fd2-9678-441fe4309da7 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:18.020856643 +0000 UTC m=+1.821802434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls") pod "image-registry-8648b894c6-t5rf2" (UID: "3ce83f95-758c-4fd2-9678-441fe4309da7") : secret "image-registry-tls" not found Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520914 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-sysctl-conf\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.520894 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520936 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-log-socket\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520966 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-cni-netd\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520967 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-log-socket\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520977 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-sysctl-d\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.520996 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-device-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521016 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-cni-netd\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521020 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-host\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521034 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-modprobe-d\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521050 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-device-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521052 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-cnibin\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521056 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-host\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521078 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-run-k8s-cni-cncf-io\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521094 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-cnibin\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521108 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-sys-fs\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521134 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-sysconfig\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521150 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-sys-fs\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521159 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-registration-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521159 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-run-k8s-cni-cncf-io\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521170 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-sysconfig\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521193 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-var-lib-cni-bin\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521199 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-registration-dir\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521218 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-etc-selinux\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521230 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-var-lib-cni-bin\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521242 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-slash\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521267 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-socket-dir-parent\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521273 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7714ee23-de1d-4e5d-88ba-cce91e08ba23-etc-selinux\") pod \"aws-ebs-csi-driver-node-lvwnl\" (UID: \"7714ee23-de1d-4e5d-88ba-cce91e08ba23\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lvwnl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521291 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-run\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521293 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-slash\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521324 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-socket-dir-parent\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521333 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-var-lib-kubelet\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521344 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-run\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521366 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-hostroot\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521394 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-etc-kubernetes\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521370 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-var-lib-kubelet\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521422 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521454 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-lib-modules\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521460 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-etc-kubernetes\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521482 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-cni-dir\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521507 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-conf-dir\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521518 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1ced90a-6b3d-4553-b05f-d4ec50e6af9b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2pkqx\" (UID: \"a1ced90a-6b3d-4553-b05f-d4ec50e6af9b\") " pod="openshift-multus/multus-additional-cni-plugins-2pkqx" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521533 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-systemd\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521543 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-hostroot\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521560 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0df87f5d-31f4-4571-82b6-fe75c434ff27-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qmh6r\" (UID: \"0df87f5d-31f4-4571-82b6-fe75c434ff27\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521574 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-cni-dir\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521586 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-lib-modules\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521587 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-multus-conf-dir\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521591 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-var-lib-kubelet\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521631 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a99c19-45c9-4294-b2af-42d20306ff28-host-var-lib-kubelet\") pod \"multus-fsbpl\" (UID: \"f7a99c19-45c9-4294-b2af-42d20306ff28\") " pod="openshift-multus/multus-fsbpl" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521615 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2d275592-23f8-4edc-aff7-016930318629-etc-systemd\") pod \"tuned-hl8q9\" (UID: \"2d275592-23f8-4edc-aff7-016930318629\") " pod="openshift-cluster-node-tuning-operator/tuned-hl8q9" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.521678 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c12382e2-90c4-45bf-bdd8-27653073cc5d-hosts-file\") pod \"node-resolver-c8xnp\" (UID: \"c12382e2-90c4-45bf-bdd8-27653073cc5d\") " pod="openshift-dns/node-resolver-c8xnp" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.522148 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c12382e2-90c4-45bf-bdd8-27653073cc5d-hosts-file\") pod \"node-resolver-c8xnp\" (UID: \"c12382e2-90c4-45bf-bdd8-27653073cc5d\") " pod="openshift-dns/node-resolver-c8xnp" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.522212 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-run-ovn\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.522238 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.522267 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-run-netns\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.522291 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26bb99fd-d223-4642-b147-b22bebd00835-dbus\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.522383 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26bb99fd-d223-4642-b147-b22bebd00835-dbus\") pod \"global-pull-secret-syncer-59fsx\" (UID: \"26bb99fd-d223-4642-b147-b22bebd00835\") " pod="kube-system/global-pull-secret-syncer-59fsx" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.522383 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab83f7b-3396-4cc7-875e-c2cc62696ff5-metrics-certs\") pod \"network-metrics-daemon-mwzdx\" (UID: \"9ab83f7b-3396-4cc7-875e-c2cc62696ff5\") " pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.522457 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.522427 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-run-ovn\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.522493 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19970c28-9dee-4377-8e98-1fc65473d3e1-host-run-netns\") pod \"ovnkube-node-vxkvk\" (UID: \"19970c28-9dee-4377-8e98-1fc65473d3e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:17.523263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.522980 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f37b227d-fc9c-4a81-85b1-62c04243ad83-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2hhp2\" (UID: \"f37b227d-fc9c-4a81-85b1-62c04243ad83\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2hhp2" Nov 04 03:49:17.526096 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.523454 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-metrics-certs\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:17.526096 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.524247 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0df87f5d-31f4-4571-82b6-fe75c434ff27-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qmh6r\" (UID: \"0df87f5d-31f4-4571-82b6-fe75c434ff27\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:17.599600 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.599572 4966 scope.go:122] "RemoveContainer" containerID="a269cc6bda79ea6e7251d9cfa350a30df870c64031267278c93cb9a0f5bd17e5" Nov 04 03:49:17.600583 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.600565 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" Nov 04 03:49:17.604989 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.604974 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2hhp2" Nov 04 03:49:17.614311 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.614259 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzdx" Nov 04 03:49:17.895984 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.895894 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r"] Nov 04 03:49:17.899217 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:17.899169 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df87f5d_31f4_4571_82b6_fe75c434ff27.slice/crio-163bf1abfaa935430ea0d57bdeb97985745a629ae2bb122ec630428d93653299 WatchSource:0}: Error finding container 163bf1abfaa935430ea0d57bdeb97985745a629ae2bb122ec630428d93653299: Status 404 returned error can't find the container with id 163bf1abfaa935430ea0d57bdeb97985745a629ae2bb122ec630428d93653299 Nov 04 03:49:17.901773 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.901752 4966 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 03:49:17.936228 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.936197 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2hhp2"] Nov 04 03:49:17.939526 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:17.939499 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf37b227d_fc9c_4a81_85b1_62c04243ad83.slice/crio-769e0b692129988a2fb3f7c8ddb91af464077248face1cd6c89804babf391cb4 WatchSource:0}: Error finding container 769e0b692129988a2fb3f7c8ddb91af464077248face1cd6c89804babf391cb4: Status 404 returned error can't find the container with id 769e0b692129988a2fb3f7c8ddb91af464077248face1cd6c89804babf391cb4 Nov 04 03:49:17.992528 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:17.992505 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mwzdx"] Nov 04 03:49:17.994991 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:17.994965 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab83f7b_3396_4cc7_875e_c2cc62696ff5.slice/crio-9d732e47090fd7f0478bb9a74a8238a5af6dd250f930ced10df19ce72c28a328 WatchSource:0}: Error finding container 9d732e47090fd7f0478bb9a74a8238a5af6dd250f930ced10df19ce72c28a328: Status 404 returned error can't find the container with id 9d732e47090fd7f0478bb9a74a8238a5af6dd250f930ced10df19ce72c28a328 Nov 04 03:49:18.036484 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:18.036449 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:18.036664 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:18.036495 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:18.036664 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:18.036589 4966 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:49:18.036664 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:18.036610 4966 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6448ff747c-dlpnw: secret "image-registry-tls" not found Nov 04 03:49:18.036664 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:18.036596 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle podName:1b81b9bc-4eb5-48a7-98e1-6c7a98117f57 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:19.036582103 +0000 UTC m=+2.837527889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle") pod "router-default-78ccf8f879-s4sx4" (UID: "1b81b9bc-4eb5-48a7-98e1-6c7a98117f57") : configmap references non-existent config key: service-ca.crt Nov 04 03:49:18.036664 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:18.036664 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:18.036900 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:18.036698 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:18.036900 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:18.036721 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:18.036900 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:18.036739 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:18.036900 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:18.036767 4966 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 03:49:18.036900 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:18.036805 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls podName:363d8742-c0a7-405c-9683-7b12afbd8bcc nodeName:}" failed. No retries permitted until 2026-11-04 03:49:19.036786542 +0000 UTC m=+2.837732332 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls") pod "image-registry-6448ff747c-dlpnw" (UID: "363d8742-c0a7-405c-9683-7b12afbd8bcc") : secret "image-registry-tls" not found Nov 04 03:49:18.036900 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:18.036818 4966 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 03:49:18.036900 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:18.036827 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls podName:7b632ca9-a695-450e-a1a9-b5357ebf4d7d nodeName:}" failed. No retries permitted until 2026-11-04 03:49:19.036818215 +0000 UTC m=+2.837764004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls") pod "dns-default-pvrd4" (UID: "7b632ca9-a695-450e-a1a9-b5357ebf4d7d") : secret "dns-default-metrics-tls" not found Nov 04 03:49:18.036900 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:18.036852 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:18.036900 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:18.036862 4966 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 03:49:18.036900 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:18.036866 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls podName:30f18911-ab5a-44e3-88b2-6508325ceb39 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:19.036852811 +0000 UTC m=+2.837798617 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6x2fz" (UID: "30f18911-ab5a-44e3-88b2-6508325ceb39") : secret "samples-operator-tls" not found Nov 04 03:49:18.037301 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:18.036932 4966 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:49:18.037301 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:18.036945 4966 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8648b894c6-t5rf2: secret "image-registry-tls" not found Nov 04 03:49:18.037301 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:18.036947 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls podName:78e55dc9-354d-455f-9d02-b3cfd5885d65 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:19.036911628 +0000 UTC m=+2.837857424 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zfjhp" (UID: "78e55dc9-354d-455f-9d02-b3cfd5885d65") : secret "insights-runtime-extractor-tls" not found Nov 04 03:49:18.037301 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:18.036987 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls podName:3ce83f95-758c-4fd2-9678-441fe4309da7 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:19.036976097 +0000 UTC m=+2.837921887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls") pod "image-registry-8648b894c6-t5rf2" (UID: "3ce83f95-758c-4fd2-9678-441fe4309da7") : secret "image-registry-tls" not found Nov 04 03:49:18.037301 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:18.037264 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78e55dc9-354d-455f-9d02-b3cfd5885d65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:18.339961 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:18.339931 4966 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:18.393639 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:18.393605 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" event={"ID":"0df87f5d-31f4-4571-82b6-fe75c434ff27","Type":"ContainerStarted","Data":"163bf1abfaa935430ea0d57bdeb97985745a629ae2bb122ec630428d93653299"} Nov 04 03:49:18.394737 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:18.394710 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mwzdx" event={"ID":"9ab83f7b-3396-4cc7-875e-c2cc62696ff5","Type":"ContainerStarted","Data":"9d732e47090fd7f0478bb9a74a8238a5af6dd250f930ced10df19ce72c28a328"} Nov 04 03:49:18.395810 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:18.395790 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2hhp2" event={"ID":"f37b227d-fc9c-4a81-85b1-62c04243ad83","Type":"ContainerStarted","Data":"769e0b692129988a2fb3f7c8ddb91af464077248face1cd6c89804babf391cb4"} Nov 04 03:49:18.397414 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:18.397378 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/2.log" Nov 04 03:49:18.397812 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:18.397797 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/1.log" Nov 04 03:49:18.397908 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:18.397832 4966 generic.go:356] "Generic (PLEG): container finished" podID="9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6" containerID="2117982782bb3a9667230e11328c05e0aa6e703f0bee1b7fb16834f879fcac0b" exitCode=255 Nov 04 03:49:18.397908 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:18.397896 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" event={"ID":"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6","Type":"ContainerDied","Data":"2117982782bb3a9667230e11328c05e0aa6e703f0bee1b7fb16834f879fcac0b"} Nov 04 03:49:18.398010 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:18.397968 4966 scope.go:122] "RemoveContainer" containerID="a269cc6bda79ea6e7251d9cfa350a30df870c64031267278c93cb9a0f5bd17e5" Nov 04 03:49:18.398209 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:18.398192 4966 scope.go:122] "RemoveContainer" containerID="2117982782bb3a9667230e11328c05e0aa6e703f0bee1b7fb16834f879fcac0b" Nov 04 03:49:18.398416 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:18.398399 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rr5bv_openshift-console-operator(9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" podUID="9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6" Nov 04 03:49:18.581503 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:18.581470 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:19.049360 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:19.049212 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:19.049360 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:19.049272 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:19.049360 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:19.049301 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:19.049360 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:19.049340 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:19.049360 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:19.049367 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:19.049641 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:19.049389 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:19.049641 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:19.049513 4966 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:49:19.049641 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:19.049524 4966 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8648b894c6-t5rf2: secret "image-registry-tls" not found Nov 04 03:49:19.049641 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:19.049568 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls podName:3ce83f95-758c-4fd2-9678-441fe4309da7 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:21.049555971 +0000 UTC m=+4.850501761 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls") pod "image-registry-8648b894c6-t5rf2" (UID: "3ce83f95-758c-4fd2-9678-441fe4309da7") : secret "image-registry-tls" not found Nov 04 03:49:19.049908 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:19.049882 4966 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:49:19.049908 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:19.049906 4966 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6448ff747c-dlpnw: secret "image-registry-tls" not found Nov 04 03:49:19.050103 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:19.049958 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls podName:363d8742-c0a7-405c-9683-7b12afbd8bcc nodeName:}" failed. No retries permitted until 2026-11-04 03:49:21.049946273 +0000 UTC m=+4.850892072 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls") pod "image-registry-6448ff747c-dlpnw" (UID: "363d8742-c0a7-405c-9683-7b12afbd8bcc") : secret "image-registry-tls" not found Nov 04 03:49:19.050103 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:19.050008 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle podName:1b81b9bc-4eb5-48a7-98e1-6c7a98117f57 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:21.050001849 +0000 UTC m=+4.850947648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle") pod "router-default-78ccf8f879-s4sx4" (UID: "1b81b9bc-4eb5-48a7-98e1-6c7a98117f57") : configmap references non-existent config key: service-ca.crt Nov 04 03:49:19.050103 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:19.050051 4966 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 03:49:19.050103 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:19.050069 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls podName:7b632ca9-a695-450e-a1a9-b5357ebf4d7d nodeName:}" failed. No retries permitted until 2026-11-04 03:49:21.050063835 +0000 UTC m=+4.851009620 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls") pod "dns-default-pvrd4" (UID: "7b632ca9-a695-450e-a1a9-b5357ebf4d7d") : secret "dns-default-metrics-tls" not found Nov 04 03:49:19.050103 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:19.050102 4966 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 03:49:19.050382 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:19.050119 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls podName:30f18911-ab5a-44e3-88b2-6508325ceb39 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:21.050113901 +0000 UTC m=+4.851059687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6x2fz" (UID: "30f18911-ab5a-44e3-88b2-6508325ceb39") : secret "samples-operator-tls" not found Nov 04 03:49:19.050382 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:19.050152 4966 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 03:49:19.050382 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:19.050178 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls podName:78e55dc9-354d-455f-9d02-b3cfd5885d65 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:21.050173104 +0000 UTC m=+4.851118889 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zfjhp" (UID: "78e55dc9-354d-455f-9d02-b3cfd5885d65") : secret "insights-runtime-extractor-tls" not found Nov 04 03:49:19.404848 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:19.404810 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2hhp2" event={"ID":"f37b227d-fc9c-4a81-85b1-62c04243ad83","Type":"ContainerStarted","Data":"889f2673abd5a3f320e1a07c63e19be23a7bdaeae8d5b1bbfc7e3cb4c1888ec9"} Nov 04 03:49:19.407343 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:19.407320 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/2.log" Nov 04 03:49:19.407649 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:19.407634 4966 scope.go:122] "RemoveContainer" containerID="2117982782bb3a9667230e11328c05e0aa6e703f0bee1b7fb16834f879fcac0b" Nov 04 03:49:19.407830 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:19.407813 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rr5bv_openshift-console-operator(9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" podUID="9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6" Nov 04 03:49:20.022327 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:20.022243 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:20.276033 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:20.275949 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:49:20.416496 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:20.416459 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mwzdx" event={"ID":"9ab83f7b-3396-4cc7-875e-c2cc62696ff5","Type":"ContainerStarted","Data":"ca43984be2fba67e6a4db22844702c228de75eadde18c4c1e18a3180d0eb9d8f"} Nov 04 03:49:20.416948 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:20.416556 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mwzdx" event={"ID":"9ab83f7b-3396-4cc7-875e-c2cc62696ff5","Type":"ContainerStarted","Data":"1f18b85272a10170ec4f6f6c573d86968d77ce7369631976a751bd7ab09bec3e"} Nov 04 03:49:20.418225 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:20.418195 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qmh6r" event={"ID":"0df87f5d-31f4-4571-82b6-fe75c434ff27","Type":"ContainerStarted","Data":"8fb9e07dd900f446ff9910571e3aeca8349c2deedbff3884123d88434c6f60d8"} Nov 04 03:49:20.418648 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:20.418630 4966 scope.go:122] "RemoveContainer" containerID="2117982782bb3a9667230e11328c05e0aa6e703f0bee1b7fb16834f879fcac0b" Nov 04 03:49:20.418811 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:20.418792 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rr5bv_openshift-console-operator(9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" podUID="9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6" Nov 04 03:49:21.070654 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:21.070618 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:21.070799 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:21.070667 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:21.070799 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:21.070693 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:21.070799 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:21.070719 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:21.070799 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:21.070779 4966 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 03:49:21.071036 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:21.070836 4966 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 03:49:21.071036 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:21.070853 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls podName:30f18911-ab5a-44e3-88b2-6508325ceb39 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:25.070836307 +0000 UTC m=+8.871782093 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6x2fz" (UID: "30f18911-ab5a-44e3-88b2-6508325ceb39") : secret "samples-operator-tls" not found Nov 04 03:49:21.071036 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:21.070857 4966 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:49:21.071036 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:21.070786 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:21.071036 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:21.070872 4966 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6448ff747c-dlpnw: secret "image-registry-tls" not found Nov 04 03:49:21.071036 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:21.070891 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls podName:78e55dc9-354d-455f-9d02-b3cfd5885d65 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:25.070876896 +0000 UTC m=+8.871822682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zfjhp" (UID: "78e55dc9-354d-455f-9d02-b3cfd5885d65") : secret "insights-runtime-extractor-tls" not found Nov 04 03:49:21.071036 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:21.070858 4966 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:49:21.071036 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:21.070904 4966 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8648b894c6-t5rf2: secret "image-registry-tls" not found Nov 04 03:49:21.071036 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:21.070935 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls podName:363d8742-c0a7-405c-9683-7b12afbd8bcc nodeName:}" failed. No retries permitted until 2026-11-04 03:49:25.070909725 +0000 UTC m=+8.871855511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls") pod "image-registry-6448ff747c-dlpnw" (UID: "363d8742-c0a7-405c-9683-7b12afbd8bcc") : secret "image-registry-tls" not found Nov 04 03:49:21.071036 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:21.070954 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls podName:3ce83f95-758c-4fd2-9678-441fe4309da7 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:25.070944754 +0000 UTC m=+8.871890546 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls") pod "image-registry-8648b894c6-t5rf2" (UID: "3ce83f95-758c-4fd2-9678-441fe4309da7") : secret "image-registry-tls" not found Nov 04 03:49:21.071036 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:21.070983 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:21.071435 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:21.071078 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle podName:1b81b9bc-4eb5-48a7-98e1-6c7a98117f57 nodeName:}" failed. No retries permitted until 2026-11-04 03:49:25.071068308 +0000 UTC m=+8.872014125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle") pod "router-default-78ccf8f879-s4sx4" (UID: "1b81b9bc-4eb5-48a7-98e1-6c7a98117f57") : configmap references non-existent config key: service-ca.crt Nov 04 03:49:21.073072 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:21.073050 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b632ca9-a695-450e-a1a9-b5357ebf4d7d-metrics-tls\") pod \"dns-default-pvrd4\" (UID: \"7b632ca9-a695-450e-a1a9-b5357ebf4d7d\") " pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:21.217466 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:21.217434 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:21.422710 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:21.422673 4966 generic.go:356] "Generic (PLEG): container finished" podID="3d313ca1-e640-42d9-a9f7-5e3e8918dc77" containerID="26eda4d6c4411a06bb40bbe3f7aba060db366a4d767a3a7310eeb374b7d4a159" exitCode=0 Nov 04 03:49:21.423146 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:21.422747 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" event={"ID":"3d313ca1-e640-42d9-a9f7-5e3e8918dc77","Type":"ContainerDied","Data":"26eda4d6c4411a06bb40bbe3f7aba060db366a4d767a3a7310eeb374b7d4a159"} Nov 04 03:49:21.423199 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:21.423147 4966 scope.go:122] "RemoveContainer" containerID="26eda4d6c4411a06bb40bbe3f7aba060db366a4d767a3a7310eeb374b7d4a159" Nov 04 03:49:21.424423 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:21.424398 4966 generic.go:356] "Generic (PLEG): container finished" podID="3b456f4f-071a-4e5f-b8b6-51c3ee790656" containerID="fd75078afffb49d9c74a8663412e6c36588f2b16945533cde860aa1d8f4d637a" exitCode=0 Nov 04 03:49:21.424544 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:21.424447 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-gm5s7" event={"ID":"3b456f4f-071a-4e5f-b8b6-51c3ee790656","Type":"ContainerDied","Data":"fd75078afffb49d9c74a8663412e6c36588f2b16945533cde860aa1d8f4d637a"} Nov 04 03:49:21.425045 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:21.425029 4966 scope.go:122] "RemoveContainer" containerID="fd75078afffb49d9c74a8663412e6c36588f2b16945533cde860aa1d8f4d637a" Nov 04 03:49:21.487869 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:21.487843 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pvrd4"] Nov 04 03:49:21.491165 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:21.491137 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b632ca9_a695_450e_a1a9_b5357ebf4d7d.slice/crio-297937c4db29b210001f41ebce65e4b01420f419942ede5daed4237e85b9e927 WatchSource:0}: Error finding container 297937c4db29b210001f41ebce65e4b01420f419942ede5daed4237e85b9e927: Status 404 returned error can't find the container with id 297937c4db29b210001f41ebce65e4b01420f419942ede5daed4237e85b9e927 Nov 04 03:49:22.431902 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:22.431382 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jgg8" event={"ID":"3d313ca1-e640-42d9-a9f7-5e3e8918dc77","Type":"ContainerStarted","Data":"326fc5fd0cfe1ced681370d141a7d265384791fd786be22e389979d98362605f"} Nov 04 03:49:22.433549 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:22.433518 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-gm5s7" event={"ID":"3b456f4f-071a-4e5f-b8b6-51c3ee790656","Type":"ContainerStarted","Data":"4190d54e93762a9c98bbfec52a01f5046a25291bc678845795a9c723304fd8b7"} Nov 04 03:49:22.434840 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:22.434813 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvrd4" event={"ID":"7b632ca9-a695-450e-a1a9-b5357ebf4d7d","Type":"ContainerStarted","Data":"297937c4db29b210001f41ebce65e4b01420f419942ede5daed4237e85b9e927"} Nov 04 03:49:23.719466 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:23.719434 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:23.719877 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:23.719665 4966 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 03:49:24.443739 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:24.443704 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvrd4" event={"ID":"7b632ca9-a695-450e-a1a9-b5357ebf4d7d","Type":"ContainerStarted","Data":"ddf7cf18e35ce6d7153c2080114233ea12d08c1e8ba238ea851916d72c57138d"} Nov 04 03:49:24.443902 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:24.443755 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvrd4" event={"ID":"7b632ca9-a695-450e-a1a9-b5357ebf4d7d","Type":"ContainerStarted","Data":"b7a748c452fe0b82ca739e0055348e117d4112fa2ebf79bfed8fea2b821c3d26"} Nov 04 03:49:24.443902 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:24.443855 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:25.107480 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.107430 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:25.107480 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.107489 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:25.108069 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.107511 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:25.108069 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.107562 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:25.108069 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.107581 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:25.108408 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.108382 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b81b9bc-4eb5-48a7-98e1-6c7a98117f57-service-ca-bundle\") pod \"router-default-78ccf8f879-s4sx4\" (UID: \"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57\") " pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:25.109898 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.109873 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78e55dc9-354d-455f-9d02-b3cfd5885d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zfjhp\" (UID: \"78e55dc9-354d-455f-9d02-b3cfd5885d65\") " pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:25.110024 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.109981 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls\") pod \"image-registry-8648b894c6-t5rf2\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:25.110322 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.110299 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls\") pod \"image-registry-6448ff747c-dlpnw\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:25.110400 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.110341 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/30f18911-ab5a-44e3-88b2-6508325ceb39-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6x2fz\" (UID: \"30f18911-ab5a-44e3-88b2-6508325ceb39\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:25.117141 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.117107 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:25.119073 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.119048 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zfjhp" Nov 04 03:49:25.399763 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.399664 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:25.400994 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.400964 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8648b894c6-t5rf2"] Nov 04 03:49:25.401116 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.401105 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:25.401881 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.401861 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" Nov 04 03:49:25.403759 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:25.403728 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce83f95_758c_4fd2_9678_441fe4309da7.slice/crio-28f134bc49d55cad184f3ba02c34f198502f96feaf1ffe84acdd01ab1f045274 WatchSource:0}: Error finding container 28f134bc49d55cad184f3ba02c34f198502f96feaf1ffe84acdd01ab1f045274: Status 404 returned error can't find the container with id 28f134bc49d55cad184f3ba02c34f198502f96feaf1ffe84acdd01ab1f045274 Nov 04 03:49:25.407072 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.407050 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zfjhp"] Nov 04 03:49:25.411733 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:25.411701 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e55dc9_354d_455f_9d02_b3cfd5885d65.slice/crio-fa2680f5f1cb107db8bb745b7241c2a081ebbf5c3389bd12e0836e43ca0e16d0 WatchSource:0}: Error finding container fa2680f5f1cb107db8bb745b7241c2a081ebbf5c3389bd12e0836e43ca0e16d0: Status 404 returned error can't find the container with id fa2680f5f1cb107db8bb745b7241c2a081ebbf5c3389bd12e0836e43ca0e16d0 Nov 04 03:49:25.461697 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.461545 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zfjhp" event={"ID":"78e55dc9-354d-455f-9d02-b3cfd5885d65","Type":"ContainerStarted","Data":"fa2680f5f1cb107db8bb745b7241c2a081ebbf5c3389bd12e0836e43ca0e16d0"} Nov 04 03:49:25.469823 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.469790 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" event={"ID":"3ce83f95-758c-4fd2-9678-441fe4309da7","Type":"ContainerStarted","Data":"28f134bc49d55cad184f3ba02c34f198502f96feaf1ffe84acdd01ab1f045274"} Nov 04 03:49:25.692847 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.692714 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6448ff747c-dlpnw"] Nov 04 03:49:25.696041 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:25.696014 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363d8742_c0a7_405c_9683_7b12afbd8bcc.slice/crio-436cc624894914b5217bb7c5420df478c944b4162d38f9326d8771c683d0a3d2 WatchSource:0}: Error finding container 436cc624894914b5217bb7c5420df478c944b4162d38f9326d8771c683d0a3d2: Status 404 returned error can't find the container with id 436cc624894914b5217bb7c5420df478c944b4162d38f9326d8771c683d0a3d2 Nov 04 03:49:25.704153 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.704123 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-78ccf8f879-s4sx4"] Nov 04 03:49:25.706210 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:25.706186 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz"] Nov 04 03:49:25.708396 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:25.708348 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b81b9bc_4eb5_48a7_98e1_6c7a98117f57.slice/crio-981e72c347b16e1caf0052658715e85db70283dd5f7652a674d878f5b31c7c6c WatchSource:0}: Error finding container 981e72c347b16e1caf0052658715e85db70283dd5f7652a674d878f5b31c7c6c: Status 404 returned error can't find the container with id 981e72c347b16e1caf0052658715e85db70283dd5f7652a674d878f5b31c7c6c Nov 04 03:49:26.276771 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:26.276743 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vxkvk" Nov 04 03:49:26.474326 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:26.474288 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" event={"ID":"363d8742-c0a7-405c-9683-7b12afbd8bcc","Type":"ContainerStarted","Data":"12dad038347c6a9c636250b2bac6024011c83e4197eb1493e808c080a3d3c245"} Nov 04 03:49:26.474513 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:26.474342 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" event={"ID":"363d8742-c0a7-405c-9683-7b12afbd8bcc","Type":"ContainerStarted","Data":"436cc624894914b5217bb7c5420df478c944b4162d38f9326d8771c683d0a3d2"} Nov 04 03:49:26.475906 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:26.475880 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zfjhp" event={"ID":"78e55dc9-354d-455f-9d02-b3cfd5885d65","Type":"ContainerStarted","Data":"770209e050755467c7ed61b8431d0f773fcd8527afa00265bd432ac6f9d84c31"} Nov 04 03:49:26.477736 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:26.477712 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" event={"ID":"3ce83f95-758c-4fd2-9678-441fe4309da7","Type":"ContainerStarted","Data":"f62d86c803b06a35a9e344215dc4ccd6a3d905008806464b6900beb649595c90"} Nov 04 03:49:26.479077 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:26.479049 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" event={"ID":"30f18911-ab5a-44e3-88b2-6508325ceb39","Type":"ContainerStarted","Data":"02af8c8e7d2709eb8a3da63b475869479cc4cff9b4fb072c629109b5c7ff900d"} Nov 04 03:49:26.480749 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:26.480714 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-78ccf8f879-s4sx4" event={"ID":"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57","Type":"ContainerStarted","Data":"87f97471d4b21377e0ba8d6996a79894b6a2e4d0fa333c5381f751454eab4ec0"} Nov 04 03:49:26.480836 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:26.480758 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-78ccf8f879-s4sx4" event={"ID":"1b81b9bc-4eb5-48a7-98e1-6c7a98117f57","Type":"ContainerStarted","Data":"981e72c347b16e1caf0052658715e85db70283dd5f7652a674d878f5b31c7c6c"} Nov 04 03:49:27.402238 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:27.402109 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:27.405207 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:27.405175 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:27.487657 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:27.487624 4966 generic.go:356] "Generic (PLEG): container finished" podID="3b456f4f-071a-4e5f-b8b6-51c3ee790656" containerID="4190d54e93762a9c98bbfec52a01f5046a25291bc678845795a9c723304fd8b7" exitCode=0 Nov 04 03:49:27.487818 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:27.487715 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-gm5s7" event={"ID":"3b456f4f-071a-4e5f-b8b6-51c3ee790656","Type":"ContainerDied","Data":"4190d54e93762a9c98bbfec52a01f5046a25291bc678845795a9c723304fd8b7"} Nov 04 03:49:27.487818 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:27.487758 4966 scope.go:122] "RemoveContainer" containerID="fd75078afffb49d9c74a8663412e6c36588f2b16945533cde860aa1d8f4d637a" Nov 04 03:49:27.488184 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:27.488163 4966 scope.go:122] "RemoveContainer" containerID="4190d54e93762a9c98bbfec52a01f5046a25291bc678845795a9c723304fd8b7" Nov 04 03:49:27.488401 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:27.488376 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-gm5s7_openshift-insights(3b456f4f-071a-4e5f-b8b6-51c3ee790656)\"" pod="openshift-insights/insights-operator-7c4986d449-gm5s7" podUID="3b456f4f-071a-4e5f-b8b6-51c3ee790656" Nov 04 03:49:27.491614 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:27.491594 4966 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 03:49:27.491981 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:27.491955 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zfjhp" event={"ID":"78e55dc9-354d-455f-9d02-b3cfd5885d65","Type":"ContainerStarted","Data":"3e45a85caa09632e62adb77878763883b9008ce0b7fac0cc25a794c27d7ba152"} Nov 04 03:49:27.492105 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:27.492063 4966 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 03:49:28.160516 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:28.160485 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:28.340807 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:28.340779 4966 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:28.341241 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:28.341223 4966 scope.go:122] "RemoveContainer" containerID="2117982782bb3a9667230e11328c05e0aa6e703f0bee1b7fb16834f879fcac0b" Nov 04 03:49:28.498265 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:28.498225 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" event={"ID":"30f18911-ab5a-44e3-88b2-6508325ceb39","Type":"ContainerStarted","Data":"f7241da151b4ee0cc3ef13888a6fb3fbc415cf720d141089f3e9def92771932e"} Nov 04 03:49:28.498716 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:28.498286 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6x2fz" event={"ID":"30f18911-ab5a-44e3-88b2-6508325ceb39","Type":"ContainerStarted","Data":"a510815c6a4abc40562f09d9ed3234aae543e038575682bb3d00c3e96b1847f0"} Nov 04 03:49:28.499789 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:28.499772 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/3.log" Nov 04 03:49:28.500172 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:28.500156 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/2.log" Nov 04 03:49:28.500209 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:28.500190 4966 generic.go:356] "Generic (PLEG): container finished" podID="9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6" containerID="48ebfc0b521b31057db5b59d99b5c089d02d0a9fc415d5a4ff429c2d4ff5b19e" exitCode=255 Nov 04 03:49:28.500282 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:28.500265 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" event={"ID":"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6","Type":"ContainerDied","Data":"48ebfc0b521b31057db5b59d99b5c089d02d0a9fc415d5a4ff429c2d4ff5b19e"} Nov 04 03:49:28.500324 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:28.500315 4966 scope.go:122] "RemoveContainer" containerID="2117982782bb3a9667230e11328c05e0aa6e703f0bee1b7fb16834f879fcac0b" Nov 04 03:49:28.500611 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:28.500596 4966 scope.go:122] "RemoveContainer" containerID="48ebfc0b521b31057db5b59d99b5c089d02d0a9fc415d5a4ff429c2d4ff5b19e" Nov 04 03:49:28.500808 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:28.500761 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rr5bv_openshift-console-operator(9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" podUID="9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6" Nov 04 03:49:28.501914 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:28.501894 4966 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 03:49:29.509497 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:29.509468 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/3.log" Nov 04 03:49:29.511790 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:29.511761 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zfjhp" event={"ID":"78e55dc9-354d-455f-9d02-b3cfd5885d65","Type":"ContainerStarted","Data":"4921adaaeef4d4bbd90d594620a2ebe25f8d452735fe204b4bee09306fdeff43"} Nov 04 03:49:30.022550 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:30.022517 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:30.022865 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:30.022852 4966 scope.go:122] "RemoveContainer" containerID="48ebfc0b521b31057db5b59d99b5c089d02d0a9fc415d5a4ff429c2d4ff5b19e" Nov 04 03:49:30.023076 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:30.023056 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rr5bv_openshift-console-operator(9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" podUID="9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6" Nov 04 03:49:30.868051 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:30.868021 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:33.194322 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:33.194282 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:33.194774 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:33.194434 4966 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 03:49:33.195630 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:33.195610 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-78ccf8f879-s4sx4" Nov 04 03:49:34.472118 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:34.472089 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pvrd4" Nov 04 03:49:35.246234 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.246187 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-hg8zs-dxlbt"] Nov 04 03:49:35.251345 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.251304 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" Nov 04 03:49:35.266037 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.266007 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hg8zs-dxlbt"] Nov 04 03:49:35.288322 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.288294 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6448ff747c-dlpnw"] Nov 04 03:49:35.300230 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.300208 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5nfh9"] Nov 04 03:49:35.303933 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.303876 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jkpht"] Nov 04 03:49:35.304058 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.304023 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5nfh9" Nov 04 03:49:35.307693 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.307667 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jkpht" Nov 04 03:49:35.308522 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.308496 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-whb4x"] Nov 04 03:49:35.311777 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.311755 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whb4x" Nov 04 03:49:35.315944 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.315435 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gwbqx\"" Nov 04 03:49:35.315944 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.315506 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 04 03:49:35.316793 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.316759 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7944j\"" Nov 04 03:49:35.317784 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.317763 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 04 03:49:35.317999 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.317978 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 04 03:49:35.317999 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.317983 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jkpht"] Nov 04 03:49:35.318791 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.318768 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5nfh9"] Nov 04 03:49:35.324863 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.324838 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-whb4x"] Nov 04 03:49:35.391350 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.391312 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5786847478-jw69k"] Nov 04 03:49:35.394504 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.394488 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.401142 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.401088 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smztx\" (UniqueName: \"kubernetes.io/projected/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-kube-api-access-smztx\") pod \"periodic-gathering-hg8zs-dxlbt\" (UID: \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\") " pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" Nov 04 03:49:35.401311 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.401138 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-archives-path\") pod \"periodic-gathering-hg8zs-dxlbt\" (UID: \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\") " pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" Nov 04 03:49:35.401311 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.401176 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-trusted-ca-bundle\") pod \"periodic-gathering-hg8zs-dxlbt\" (UID: \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\") " pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" Nov 04 03:49:35.401432 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.401339 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ae821c43-9b3a-408a-8090-82ff08946607-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5nfh9\" (UID: \"ae821c43-9b3a-408a-8090-82ff08946607\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5nfh9" Nov 04 03:49:35.401479 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.401445 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-service-ca-bundle\") pod \"periodic-gathering-hg8zs-dxlbt\" (UID: \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\") " pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" Nov 04 03:49:35.401559 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.401536 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvlb9\" (UniqueName: \"kubernetes.io/projected/a9c51069-ff47-4267-ad60-9ac364b7face-kube-api-access-bvlb9\") pod \"migrator-66df5d496b-jkpht\" (UID: \"a9c51069-ff47-4267-ad60-9ac364b7face\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jkpht" Nov 04 03:49:35.428128 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.428102 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5786847478-jw69k"] Nov 04 03:49:35.502760 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.502678 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7n9s\" (UniqueName: \"kubernetes.io/projected/92a52dcc-746c-4c78-91fb-857ecf2dfd0e-kube-api-access-f7n9s\") pod \"migrator-66df5d496b-whb4x\" (UID: \"92a52dcc-746c-4c78-91fb-857ecf2dfd0e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whb4x" Nov 04 03:49:35.503349 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.503319 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e4cc7333-a66b-42e0-a925-7a3be02d606e-ca-trust-extracted\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.503485 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.503467 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e4cc7333-a66b-42e0-a925-7a3be02d606e-installation-pull-secrets\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.503600 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.503586 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smztx\" (UniqueName: \"kubernetes.io/projected/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-kube-api-access-smztx\") pod \"periodic-gathering-hg8zs-dxlbt\" (UID: \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\") " pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" Nov 04 03:49:35.503725 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.503709 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e4cc7333-a66b-42e0-a925-7a3be02d606e-image-registry-private-configuration\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.503847 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.503833 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4cc7333-a66b-42e0-a925-7a3be02d606e-bound-sa-token\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.503983 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.503968 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-archives-path\") pod \"periodic-gathering-hg8zs-dxlbt\" (UID: \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\") " pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" Nov 04 03:49:35.504117 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.504103 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-trusted-ca-bundle\") pod \"periodic-gathering-hg8zs-dxlbt\" (UID: \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\") " pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" Nov 04 03:49:35.504216 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.504203 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ae821c43-9b3a-408a-8090-82ff08946607-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5nfh9\" (UID: \"ae821c43-9b3a-408a-8090-82ff08946607\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5nfh9" Nov 04 03:49:35.504319 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.504306 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-service-ca-bundle\") pod \"periodic-gathering-hg8zs-dxlbt\" (UID: \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\") " pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" Nov 04 03:49:35.504423 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.504410 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvlb9\" (UniqueName: \"kubernetes.io/projected/a9c51069-ff47-4267-ad60-9ac364b7face-kube-api-access-bvlb9\") pod \"migrator-66df5d496b-jkpht\" (UID: \"a9c51069-ff47-4267-ad60-9ac364b7face\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jkpht" Nov 04 03:49:35.504524 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.504511 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e4cc7333-a66b-42e0-a925-7a3be02d606e-trusted-ca\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.505254 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.505230 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-archives-path\") pod \"periodic-gathering-hg8zs-dxlbt\" (UID: \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\") " pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" Nov 04 03:49:35.505549 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.505524 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e4cc7333-a66b-42e0-a925-7a3be02d606e-registry-certificates\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.506774 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.506749 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-trusted-ca-bundle\") pod \"periodic-gathering-hg8zs-dxlbt\" (UID: \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\") " pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" Nov 04 03:49:35.507390 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.507360 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwrr4\" (UniqueName: \"kubernetes.io/projected/e4cc7333-a66b-42e0-a925-7a3be02d606e-kube-api-access-rwrr4\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.507759 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.507727 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e4cc7333-a66b-42e0-a925-7a3be02d606e-registry-tls\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.507964 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.507941 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-service-ca-bundle\") pod \"periodic-gathering-hg8zs-dxlbt\" (UID: \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\") " pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" Nov 04 03:49:35.508159 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.508140 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ae821c43-9b3a-408a-8090-82ff08946607-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5nfh9\" (UID: \"ae821c43-9b3a-408a-8090-82ff08946607\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5nfh9" Nov 04 03:49:35.517526 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.517495 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvlb9\" (UniqueName: \"kubernetes.io/projected/a9c51069-ff47-4267-ad60-9ac364b7face-kube-api-access-bvlb9\") pod \"migrator-66df5d496b-jkpht\" (UID: \"a9c51069-ff47-4267-ad60-9ac364b7face\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jkpht" Nov 04 03:49:35.518209 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.518184 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smztx\" (UniqueName: \"kubernetes.io/projected/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-kube-api-access-smztx\") pod \"periodic-gathering-hg8zs-dxlbt\" (UID: \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\") " pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" Nov 04 03:49:35.562974 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.562949 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" Nov 04 03:49:35.608557 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.608518 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7n9s\" (UniqueName: \"kubernetes.io/projected/92a52dcc-746c-4c78-91fb-857ecf2dfd0e-kube-api-access-f7n9s\") pod \"migrator-66df5d496b-whb4x\" (UID: \"92a52dcc-746c-4c78-91fb-857ecf2dfd0e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whb4x" Nov 04 03:49:35.608733 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.608572 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e4cc7333-a66b-42e0-a925-7a3be02d606e-ca-trust-extracted\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.608733 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.608597 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e4cc7333-a66b-42e0-a925-7a3be02d606e-installation-pull-secrets\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.608733 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.608636 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e4cc7333-a66b-42e0-a925-7a3be02d606e-image-registry-private-configuration\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.608733 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.608666 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4cc7333-a66b-42e0-a925-7a3be02d606e-bound-sa-token\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.608733 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.608721 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e4cc7333-a66b-42e0-a925-7a3be02d606e-trusted-ca\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.609056 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.608752 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e4cc7333-a66b-42e0-a925-7a3be02d606e-registry-certificates\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.609056 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.608796 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwrr4\" (UniqueName: \"kubernetes.io/projected/e4cc7333-a66b-42e0-a925-7a3be02d606e-kube-api-access-rwrr4\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.609056 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.608821 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e4cc7333-a66b-42e0-a925-7a3be02d606e-registry-tls\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.610449 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.610374 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e4cc7333-a66b-42e0-a925-7a3be02d606e-ca-trust-extracted\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.611589 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.611562 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e4cc7333-a66b-42e0-a925-7a3be02d606e-registry-certificates\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.612177 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.612065 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e4cc7333-a66b-42e0-a925-7a3be02d606e-trusted-ca\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.619018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.615151 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e4cc7333-a66b-42e0-a925-7a3be02d606e-installation-pull-secrets\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.619018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.617224 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e4cc7333-a66b-42e0-a925-7a3be02d606e-registry-tls\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.619018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.617957 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5nfh9" Nov 04 03:49:35.619018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.618942 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e4cc7333-a66b-42e0-a925-7a3be02d606e-image-registry-private-configuration\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.621728 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.621428 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7n9s\" (UniqueName: \"kubernetes.io/projected/92a52dcc-746c-4c78-91fb-857ecf2dfd0e-kube-api-access-f7n9s\") pod \"migrator-66df5d496b-whb4x\" (UID: \"92a52dcc-746c-4c78-91fb-857ecf2dfd0e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whb4x" Nov 04 03:49:35.621728 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.621690 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4cc7333-a66b-42e0-a925-7a3be02d606e-bound-sa-token\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.624929 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.624876 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwrr4\" (UniqueName: \"kubernetes.io/projected/e4cc7333-a66b-42e0-a925-7a3be02d606e-kube-api-access-rwrr4\") pod \"image-registry-5786847478-jw69k\" (UID: \"e4cc7333-a66b-42e0-a925-7a3be02d606e\") " pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.629392 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.629369 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jkpht" Nov 04 03:49:35.634526 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.634509 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whb4x" Nov 04 03:49:35.685442 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:35.685275 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:49:35.708965 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.708911 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:35.834676 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.834591 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hg8zs-dxlbt"] Nov 04 03:49:35.837181 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:35.837151 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f9c9c6_4fae_443b_bd39_ce8b75bdb009.slice/crio-014fd36d2cb2ac0775cb8768f7a3c65efc9aac096eb928fdc7946d23c2eb6693 WatchSource:0}: Error finding container 014fd36d2cb2ac0775cb8768f7a3c65efc9aac096eb928fdc7946d23c2eb6693: Status 404 returned error can't find the container with id 014fd36d2cb2ac0775cb8768f7a3c65efc9aac096eb928fdc7946d23c2eb6693 Nov 04 03:49:35.906343 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.906317 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5nfh9"] Nov 04 03:49:35.908878 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:35.908854 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae821c43_9b3a_408a_8090_82ff08946607.slice/crio-40e84a3a4876549b498e2acd05e617cbfbd2d8301eeff8e1e5317e26240cc85b WatchSource:0}: Error finding container 40e84a3a4876549b498e2acd05e617cbfbd2d8301eeff8e1e5317e26240cc85b: Status 404 returned error can't find the container with id 40e84a3a4876549b498e2acd05e617cbfbd2d8301eeff8e1e5317e26240cc85b Nov 04 03:49:35.940419 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.940374 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jkpht"] Nov 04 03:49:35.943145 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.943119 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-whb4x"] Nov 04 03:49:35.944103 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:35.944073 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c51069_ff47_4267_ad60_9ac364b7face.slice/crio-f8fb0a0b84afa2bd71b5edecc44b3354c4a22a88d1497b4bda2c3f555694b104 WatchSource:0}: Error finding container f8fb0a0b84afa2bd71b5edecc44b3354c4a22a88d1497b4bda2c3f555694b104: Status 404 returned error can't find the container with id f8fb0a0b84afa2bd71b5edecc44b3354c4a22a88d1497b4bda2c3f555694b104 Nov 04 03:49:35.947467 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:35.947441 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a52dcc_746c_4c78_91fb_857ecf2dfd0e.slice/crio-941d8b8826b1ca299fdc919fd0456b9ca621111d533297b4ba6b970b53e11a80 WatchSource:0}: Error finding container 941d8b8826b1ca299fdc919fd0456b9ca621111d533297b4ba6b970b53e11a80: Status 404 returned error can't find the container with id 941d8b8826b1ca299fdc919fd0456b9ca621111d533297b4ba6b970b53e11a80 Nov 04 03:49:35.996432 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:35.996407 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5786847478-jw69k"] Nov 04 03:49:35.999347 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:35.999322 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cc7333_a66b_42e0_a925_7a3be02d606e.slice/crio-a0f2255a48a4eb924929ce893f3754d7c9f372ab649a55d1f7da0e8d02b75726 WatchSource:0}: Error finding container a0f2255a48a4eb924929ce893f3754d7c9f372ab649a55d1f7da0e8d02b75726: Status 404 returned error can't find the container with id a0f2255a48a4eb924929ce893f3754d7c9f372ab649a55d1f7da0e8d02b75726 Nov 04 03:49:36.539647 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:36.539608 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" event={"ID":"09f9c9c6-4fae-443b-bd39-ce8b75bdb009","Type":"ContainerStarted","Data":"0ce563f3cf9ff64f2130a79e1831baa3bb63bbdba255532dbb32311dd8719f24"} Nov 04 03:49:36.540173 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:36.539664 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" event={"ID":"09f9c9c6-4fae-443b-bd39-ce8b75bdb009","Type":"ContainerStarted","Data":"014fd36d2cb2ac0775cb8768f7a3c65efc9aac096eb928fdc7946d23c2eb6693"} Nov 04 03:49:36.541280 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:36.541239 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jkpht" event={"ID":"a9c51069-ff47-4267-ad60-9ac364b7face","Type":"ContainerStarted","Data":"f8fb0a0b84afa2bd71b5edecc44b3354c4a22a88d1497b4bda2c3f555694b104"} Nov 04 03:49:36.543371 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:36.543346 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5786847478-jw69k" event={"ID":"e4cc7333-a66b-42e0-a925-7a3be02d606e","Type":"ContainerStarted","Data":"b40801e1438f4bcad30df49dd9e9bc9816afca3237b1d98630817aa0c26924e8"} Nov 04 03:49:36.543482 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:36.543394 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5786847478-jw69k" event={"ID":"e4cc7333-a66b-42e0-a925-7a3be02d606e","Type":"ContainerStarted","Data":"a0f2255a48a4eb924929ce893f3754d7c9f372ab649a55d1f7da0e8d02b75726"} Nov 04 03:49:36.543482 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:36.543433 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:36.545065 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:36.545043 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whb4x" event={"ID":"92a52dcc-746c-4c78-91fb-857ecf2dfd0e","Type":"ContainerStarted","Data":"941d8b8826b1ca299fdc919fd0456b9ca621111d533297b4ba6b970b53e11a80"} Nov 04 03:49:36.546316 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:36.546293 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5nfh9" event={"ID":"ae821c43-9b3a-408a-8090-82ff08946607","Type":"ContainerStarted","Data":"40e84a3a4876549b498e2acd05e617cbfbd2d8301eeff8e1e5317e26240cc85b"} Nov 04 03:49:36.560300 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:36.560256 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" podStartSLOduration=1.5602438269999999 podStartE2EDuration="1.560243827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:36.55911615 +0000 UTC m=+20.360061960" watchObservedRunningTime="2026-11-04 03:49:36.560243827 +0000 UTC m=+20.361189635" Nov 04 03:49:37.333949 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:37.333901 4966 scope.go:122] "RemoveContainer" containerID="4190d54e93762a9c98bbfec52a01f5046a25291bc678845795a9c723304fd8b7" Nov 04 03:49:37.384379 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:37.384323 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5786847478-jw69k" podStartSLOduration=2.384288433 podStartE2EDuration="2.384288433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:36.584948085 +0000 UTC m=+20.385893894" watchObservedRunningTime="2026-11-04 03:49:37.384288433 +0000 UTC m=+21.185234253" Nov 04 03:49:37.485291 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:37.485259 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pvrd4_7b632ca9-a695-450e-a1a9-b5357ebf4d7d/dns/0.log" Nov 04 03:49:37.556461 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:37.556421 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-gm5s7" event={"ID":"3b456f4f-071a-4e5f-b8b6-51c3ee790656","Type":"ContainerStarted","Data":"34d9bee2e6088240276969c307da187fbd15695a5e8a0b98004833eadec5c1d1"} Nov 04 03:49:37.657567 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:37.657418 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pvrd4_7b632ca9-a695-450e-a1a9-b5357ebf4d7d/kube-rbac-proxy/0.log" Nov 04 03:49:38.055353 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:38.055327 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c8xnp_c12382e2-90c4-45bf-bdd8-27653073cc5d/dns-node-resolver/0.log" Nov 04 03:49:38.340872 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:38.340842 4966 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:38.341233 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:38.341220 4966 scope.go:122] "RemoveContainer" containerID="48ebfc0b521b31057db5b59d99b5c089d02d0a9fc415d5a4ff429c2d4ff5b19e" Nov 04 03:49:38.341413 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:49:38.341395 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rr5bv_openshift-console-operator(9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" podUID="9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6" Nov 04 03:49:38.459861 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:38.459835 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5786847478-jw69k_e4cc7333-a66b-42e0-a925-7a3be02d606e/registry/0.log" Nov 04 03:49:38.561089 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:38.561050 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whb4x" event={"ID":"92a52dcc-746c-4c78-91fb-857ecf2dfd0e","Type":"ContainerStarted","Data":"86da1a3100a9baae10f30af040c2adcb2e08c40fd3a7a3e804d25612408be59e"} Nov 04 03:49:38.562590 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:38.562566 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5nfh9" event={"ID":"ae821c43-9b3a-408a-8090-82ff08946607","Type":"ContainerStarted","Data":"1a949c867e7dd69dd5d08a64c48e3c0fdcd4536ed2c626d493b78d3e36a3fd75"} Nov 04 03:49:38.562713 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:38.562618 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5nfh9" Nov 04 03:49:38.564149 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:38.564127 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jkpht" event={"ID":"a9c51069-ff47-4267-ad60-9ac364b7face","Type":"ContainerStarted","Data":"012c5428bd6bc55f12a52cc9791498312dfe323b65cb49c09548e61aa3bb322a"} Nov 04 03:49:38.568012 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:38.567994 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5nfh9" Nov 04 03:49:38.580356 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:38.580320 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whb4x" podStartSLOduration=1.655513279 podStartE2EDuration="3.580307511s" totalImagesPullingTime="1.924794232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:38.579254574 +0000 UTC m=+22.380200383" watchObservedRunningTime="2026-11-04 03:49:38.580307511 +0000 UTC m=+22.381253315" Nov 04 03:49:38.598851 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:38.598784 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5nfh9" podStartSLOduration=1.590749526 podStartE2EDuration="3.59877247s" totalImagesPullingTime="2.008022944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:38.597689674 +0000 UTC m=+22.398635481" watchObservedRunningTime="2026-11-04 03:49:38.59877247 +0000 UTC m=+22.399718274" Nov 04 03:49:38.655362 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:38.655336 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6448ff747c-dlpnw_363d8742-c0a7-405c-9683-7b12afbd8bcc/registry/0.log" Nov 04 03:49:38.858386 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:38.858303 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8648b894c6-t5rf2_3ce83f95-758c-4fd2-9678-441fe4309da7/registry/0.log" Nov 04 03:49:39.455961 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:39.455935 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jkpjj_795d3139-8d19-4852-bd9c-3c60b01a97f6/node-ca/0.log" Nov 04 03:49:39.656888 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:39.656863 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78ccf8f879-s4sx4_1b81b9bc-4eb5-48a7-98e1-6c7a98117f57/router/0.log" Nov 04 03:49:39.856077 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:39.856050 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-chk4t_bcc0d689-48b6-467e-bdd0-6a03878a52b6/serve-healthcheck-canary/0.log" Nov 04 03:49:40.056493 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:40.056461 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-jkpht_a9c51069-ff47-4267-ad60-9ac364b7face/migrator/0.log" Nov 04 03:49:40.257018 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:40.256944 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-whb4x_92a52dcc-746c-4c78-91fb-857ecf2dfd0e/migrator/0.log" Nov 04 03:49:40.457039 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:40.457008 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm_d2052e7f-5ee8-4066-bb29-cd227fa0965b/kube-storage-version-migrator-operator/0.log" Nov 04 03:49:45.172661 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.172612 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jkpht" podStartSLOduration=8.163002913 podStartE2EDuration="10.172597049s" totalImagesPullingTime="2.009594136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:38.630734845 +0000 UTC m=+22.431680653" watchObservedRunningTime="2026-11-04 03:49:45.172597049 +0000 UTC m=+28.973542889" Nov 04 03:49:45.174335 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.174308 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mnzz5"] Nov 04 03:49:45.202381 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.202357 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.229103 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.229070 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 04 03:49:45.230113 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.230090 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rh4kv\"" Nov 04 03:49:45.230113 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.230105 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 04 03:49:45.256298 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.256272 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 04 03:49:45.256406 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.256293 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 04 03:49:45.280751 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.280727 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-sys\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.280863 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.280764 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-node-exporter-textfile\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.280863 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.280790 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.280992 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.280868 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-node-exporter-wtmp\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.280992 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.280972 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-root\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.281091 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.281002 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-node-exporter-tls\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.281132 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.281091 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.281169 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.281136 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92thx\" (UniqueName: \"kubernetes.io/projected/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-kube-api-access-92thx\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.281208 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.281181 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-metrics-client-ca\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.294446 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.294427 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:49:45.382499 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.382466 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-root\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.382639 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.382514 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-node-exporter-tls\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.382639 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.382559 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.382639 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.382567 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-root\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.382639 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.382595 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92thx\" (UniqueName: \"kubernetes.io/projected/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-kube-api-access-92thx\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.382829 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.382637 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-metrics-client-ca\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.382829 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.382685 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-sys\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.382829 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.382711 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-node-exporter-textfile\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.382829 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.382738 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.382829 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.382772 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-node-exporter-wtmp\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.383095 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.382991 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-node-exporter-wtmp\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.383095 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.383059 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-sys\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.383239 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.383216 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-node-exporter-textfile\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.383349 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.383329 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-metrics-client-ca\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.383546 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.383527 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.384912 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.384889 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.385111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.385089 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-node-exporter-tls\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.395003 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.394980 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92thx\" (UniqueName: \"kubernetes.io/projected/a2706ae5-dd36-4a7c-ba89-cf0a497f92b0-kube-api-access-92thx\") pod \"node-exporter-mnzz5\" (UID: \"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0\") " pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.570690 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:45.570658 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mnzz5" Nov 04 03:49:45.617151 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:49:45.617121 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2706ae5_dd36_4a7c_ba89_cf0a497f92b0.slice/crio-a833c64559deb3e9540be2840751f542460ddb40bff167d00f976760aee46df0 WatchSource:0}: Error finding container a833c64559deb3e9540be2840751f542460ddb40bff167d00f976760aee46df0: Status 404 returned error can't find the container with id a833c64559deb3e9540be2840751f542460ddb40bff167d00f976760aee46df0 Nov 04 03:49:46.603131 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:46.603098 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mnzz5" event={"ID":"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0","Type":"ContainerStarted","Data":"a833c64559deb3e9540be2840751f542460ddb40bff167d00f976760aee46df0"} Nov 04 03:49:47.127079 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:47.127051 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jvqw2" Nov 04 03:49:47.608904 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:47.608866 4966 generic.go:356] "Generic (PLEG): container finished" podID="a2706ae5-dd36-4a7c-ba89-cf0a497f92b0" containerID="9b7729af79ad2f7faa7013c5f87bdf9c78731ae8b41a7b8e81003e4e04cd2e99" exitCode=0 Nov 04 03:49:47.609284 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:47.608962 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mnzz5" event={"ID":"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0","Type":"ContainerDied","Data":"9b7729af79ad2f7faa7013c5f87bdf9c78731ae8b41a7b8e81003e4e04cd2e99"} Nov 04 03:49:48.505883 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:48.505856 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:49:48.616123 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:48.616080 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mnzz5" event={"ID":"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0","Type":"ContainerStarted","Data":"c8fa6cc69fb78be8e3872343e5086466fce1cc3d94be9d95a4adb05b8bffaa9d"} Nov 04 03:49:48.616489 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:48.616140 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mnzz5" event={"ID":"a2706ae5-dd36-4a7c-ba89-cf0a497f92b0","Type":"ContainerStarted","Data":"c1ffaf016544f37f6dee635764f90dd56443dfaba1efa36a8d4a08ff44a24900"} Nov 04 03:49:48.640128 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:48.640075 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mnzz5" podStartSLOduration=1.9405085359999998 podStartE2EDuration="3.640058322s" totalImagesPullingTime="1.621859458s" totalInitContainerRuntime="77.690328ms" isStatefulPod=true podCreationTimestamp="2026-11-04 03:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:49:48.638649859 +0000 UTC m=+32.439595667" watchObservedRunningTime="2026-11-04 03:49:48.640058322 +0000 UTC m=+32.441004125" Nov 04 03:49:50.278709 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:50.278647 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pbsd4" Nov 04 03:49:50.333937 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:50.333896 4966 scope.go:122] "RemoveContainer" containerID="48ebfc0b521b31057db5b59d99b5c089d02d0a9fc415d5a4ff429c2d4ff5b19e" Nov 04 03:49:50.626802 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:50.626768 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/3.log" Nov 04 03:49:50.626979 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:50.626889 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" event={"ID":"9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6","Type":"ContainerStarted","Data":"fcaa48c282acf732224acd0a589dcda29b9c3e365bc0b4f458f3e99fc871adca"} Nov 04 03:49:50.627213 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:50.627191 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:50.865797 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:50.865744 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rr5bv" Nov 04 03:49:57.562346 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:57.562317 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5786847478-jw69k" Nov 04 03:49:57.618025 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:49:57.617990 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8648b894c6-t5rf2"] Nov 04 03:50:00.309828 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.309735 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" podUID="363d8742-c0a7-405c-9683-7b12afbd8bcc" containerName="registry" containerID="cri-o://12dad038347c6a9c636250b2bac6024011c83e4197eb1493e808c080a3d3c245" gracePeriod=30 Nov 04 03:50:00.565513 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.565465 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:50:00.623189 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.623161 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-bound-sa-token\") pod \"363d8742-c0a7-405c-9683-7b12afbd8bcc\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " Nov 04 03:50:00.623316 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.623199 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-installation-pull-secrets\") pod \"363d8742-c0a7-405c-9683-7b12afbd8bcc\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " Nov 04 03:50:00.623316 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.623227 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-certificates\") pod \"363d8742-c0a7-405c-9683-7b12afbd8bcc\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " Nov 04 03:50:00.623316 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.623272 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-kube-api-access-952rw\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-kube-api-access-952rw\") pod \"363d8742-c0a7-405c-9683-7b12afbd8bcc\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " Nov 04 03:50:00.623483 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.623333 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363d8742-c0a7-405c-9683-7b12afbd8bcc-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/363d8742-c0a7-405c-9683-7b12afbd8bcc-ca-trust-extracted\") pod \"363d8742-c0a7-405c-9683-7b12afbd8bcc\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " Nov 04 03:50:00.623483 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.623396 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-image-registry-private-configuration\") pod \"363d8742-c0a7-405c-9683-7b12afbd8bcc\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " Nov 04 03:50:00.623483 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.623430 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-trusted-ca\") pod \"363d8742-c0a7-405c-9683-7b12afbd8bcc\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " Nov 04 03:50:00.623644 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.623502 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls\") pod \"363d8742-c0a7-405c-9683-7b12afbd8bcc\" (UID: \"363d8742-c0a7-405c-9683-7b12afbd8bcc\") " Nov 04 03:50:00.623840 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.623810 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-certificates" pod "363d8742-c0a7-405c-9683-7b12afbd8bcc" (UID: "363d8742-c0a7-405c-9683-7b12afbd8bcc"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:50:00.624016 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.623903 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-trusted-ca" pod "363d8742-c0a7-405c-9683-7b12afbd8bcc" (UID: "363d8742-c0a7-405c-9683-7b12afbd8bcc"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:50:00.626222 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.626193 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-installation-pull-secrets" pod "363d8742-c0a7-405c-9683-7b12afbd8bcc" (UID: "363d8742-c0a7-405c-9683-7b12afbd8bcc"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:50:00.626336 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.626248 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-bound-sa-token" pod "363d8742-c0a7-405c-9683-7b12afbd8bcc" (UID: "363d8742-c0a7-405c-9683-7b12afbd8bcc"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:50:00.626336 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.626327 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls" pod "363d8742-c0a7-405c-9683-7b12afbd8bcc" (UID: "363d8742-c0a7-405c-9683-7b12afbd8bcc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:50:00.626431 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.626330 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-kube-api-access-952rw" pod "363d8742-c0a7-405c-9683-7b12afbd8bcc" (UID: "363d8742-c0a7-405c-9683-7b12afbd8bcc"). InnerVolumeSpecName "kube-api-access-952rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:50:00.626431 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.626342 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-image-registry-private-configuration" pod "363d8742-c0a7-405c-9683-7b12afbd8bcc" (UID: "363d8742-c0a7-405c-9683-7b12afbd8bcc"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:50:00.632378 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.632347 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363d8742-c0a7-405c-9683-7b12afbd8bcc-ca-trust-extracted" pod "363d8742-c0a7-405c-9683-7b12afbd8bcc" (UID: "363d8742-c0a7-405c-9683-7b12afbd8bcc"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:50:00.675701 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.675676 4966 generic.go:356] "Generic (PLEG): container finished" podID="363d8742-c0a7-405c-9683-7b12afbd8bcc" containerID="12dad038347c6a9c636250b2bac6024011c83e4197eb1493e808c080a3d3c245" exitCode=0 Nov 04 03:50:00.675831 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.675721 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" event={"ID":"363d8742-c0a7-405c-9683-7b12afbd8bcc","Type":"ContainerDied","Data":"12dad038347c6a9c636250b2bac6024011c83e4197eb1493e808c080a3d3c245"} Nov 04 03:50:00.675831 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.675729 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" Nov 04 03:50:00.675831 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.675782 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6448ff747c-dlpnw" event={"ID":"363d8742-c0a7-405c-9683-7b12afbd8bcc","Type":"ContainerDied","Data":"436cc624894914b5217bb7c5420df478c944b4162d38f9326d8771c683d0a3d2"} Nov 04 03:50:00.675831 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.675801 4966 scope.go:122] "RemoveContainer" containerID="12dad038347c6a9c636250b2bac6024011c83e4197eb1493e808c080a3d3c245" Nov 04 03:50:00.684052 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.684032 4966 scope.go:122] "RemoveContainer" containerID="12dad038347c6a9c636250b2bac6024011c83e4197eb1493e808c080a3d3c245" Nov 04 03:50:00.684332 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:50:00.684309 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12dad038347c6a9c636250b2bac6024011c83e4197eb1493e808c080a3d3c245\": container with ID starting with 12dad038347c6a9c636250b2bac6024011c83e4197eb1493e808c080a3d3c245 not found: ID does not exist" containerID="12dad038347c6a9c636250b2bac6024011c83e4197eb1493e808c080a3d3c245" Nov 04 03:50:00.684403 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.684344 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12dad038347c6a9c636250b2bac6024011c83e4197eb1493e808c080a3d3c245"} err="failed to get container status \"12dad038347c6a9c636250b2bac6024011c83e4197eb1493e808c080a3d3c245\": rpc error: code = NotFound desc = could not find container \"12dad038347c6a9c636250b2bac6024011c83e4197eb1493e808c080a3d3c245\": container with ID starting with 12dad038347c6a9c636250b2bac6024011c83e4197eb1493e808c080a3d3c245 not found: ID does not exist" Nov 04 03:50:00.697310 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.697291 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6448ff747c-dlpnw"] Nov 04 03:50:00.707502 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.707481 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6448ff747c-dlpnw"] Nov 04 03:50:00.724595 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.724573 4966 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-tls\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:00.724595 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.724595 4966 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-bound-sa-token\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:00.724734 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.724608 4966 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-installation-pull-secrets\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:00.724734 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.724618 4966 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-registry-certificates\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:00.724734 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.724627 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-952rw\" (UniqueName: \"kubernetes.io/projected/363d8742-c0a7-405c-9683-7b12afbd8bcc-kube-api-access-952rw\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:00.724734 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.724635 4966 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/363d8742-c0a7-405c-9683-7b12afbd8bcc-ca-trust-extracted\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:00.724734 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.724650 4966 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/363d8742-c0a7-405c-9683-7b12afbd8bcc-image-registry-private-configuration\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:00.724734 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:00.724665 4966 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/363d8742-c0a7-405c-9683-7b12afbd8bcc-trusted-ca\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:02.339998 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:02.339965 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363d8742-c0a7-405c-9683-7b12afbd8bcc" path="/var/lib/kubelet/pods/363d8742-c0a7-405c-9683-7b12afbd8bcc/volumes" Nov 04 03:50:05.676334 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:50:05.676300 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:50:07.578593 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.578547 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-hg8zs-dxlbt"] Nov 04 03:50:07.579134 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.578739 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" podUID="09f9c9c6-4fae-443b-bd39-ce8b75bdb009" containerName="insights-gathering" containerID="cri-o://0ce563f3cf9ff64f2130a79e1831baa3bb63bbdba255532dbb32311dd8719f24" gracePeriod=30 Nov 04 03:50:07.582652 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.582626 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-xwdp2-kwf4s"] Nov 04 03:50:07.582824 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.582793 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" podUID="341692ff-c7ac-4387-b9a3-d41ddd8f2ad3" containerName="insights-gathering" containerID="cri-o://0d406fb7c7275db00231e338cea8ff883fb5db9637204678bfffa553cf628eee" gracePeriod=30 Nov 04 03:50:07.707101 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.707067 4966 generic.go:356] "Generic (PLEG): container finished" podID="09f9c9c6-4fae-443b-bd39-ce8b75bdb009" containerID="0ce563f3cf9ff64f2130a79e1831baa3bb63bbdba255532dbb32311dd8719f24" exitCode=2 Nov 04 03:50:07.707220 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.707137 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" event={"ID":"09f9c9c6-4fae-443b-bd39-ce8b75bdb009","Type":"ContainerDied","Data":"0ce563f3cf9ff64f2130a79e1831baa3bb63bbdba255532dbb32311dd8719f24"} Nov 04 03:50:07.708945 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.708905 4966 generic.go:356] "Generic (PLEG): container finished" podID="341692ff-c7ac-4387-b9a3-d41ddd8f2ad3" containerID="0d406fb7c7275db00231e338cea8ff883fb5db9637204678bfffa553cf628eee" exitCode=2 Nov 04 03:50:07.709060 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.708953 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" event={"ID":"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3","Type":"ContainerDied","Data":"0d406fb7c7275db00231e338cea8ff883fb5db9637204678bfffa553cf628eee"} Nov 04 03:50:07.840789 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.840735 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" Nov 04 03:50:07.877396 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.877370 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:50:07.988133 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.988105 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-service-ca-bundle\") pod \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\" (UID: \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\") " Nov 04 03:50:07.988295 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.988178 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-service-ca-bundle\") pod \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " Nov 04 03:50:07.988295 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.988209 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-archives-path\") pod \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\" (UID: \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\") " Nov 04 03:50:07.988295 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.988227 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-archives-path\") pod \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " Nov 04 03:50:07.988295 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.988256 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-trusted-ca-bundle\") pod \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\" (UID: \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\") " Nov 04 03:50:07.988295 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.988294 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-kube-api-access-lsvdh\" (UniqueName: \"kubernetes.io/projected/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-kube-api-access-lsvdh\") pod \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " Nov 04 03:50:07.988629 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.988361 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-trusted-ca-bundle\") pod \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\" (UID: \"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3\") " Nov 04 03:50:07.988629 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.988388 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-kube-api-access-smztx\" (UniqueName: \"kubernetes.io/projected/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-kube-api-access-smztx\") pod \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\" (UID: \"09f9c9c6-4fae-443b-bd39-ce8b75bdb009\") " Nov 04 03:50:07.988629 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.988517 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-service-ca-bundle" pod "09f9c9c6-4fae-443b-bd39-ce8b75bdb009" (UID: "09f9c9c6-4fae-443b-bd39-ce8b75bdb009"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:50:07.988629 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.988575 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-archives-path" pod "09f9c9c6-4fae-443b-bd39-ce8b75bdb009" (UID: "09f9c9c6-4fae-443b-bd39-ce8b75bdb009"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:50:07.988629 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.988578 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-service-ca-bundle" pod "341692ff-c7ac-4387-b9a3-d41ddd8f2ad3" (UID: "341692ff-c7ac-4387-b9a3-d41ddd8f2ad3"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:50:07.988629 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.988594 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-archives-path" pod "341692ff-c7ac-4387-b9a3-d41ddd8f2ad3" (UID: "341692ff-c7ac-4387-b9a3-d41ddd8f2ad3"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:50:07.988888 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.988660 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-trusted-ca-bundle" pod "09f9c9c6-4fae-443b-bd39-ce8b75bdb009" (UID: "09f9c9c6-4fae-443b-bd39-ce8b75bdb009"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:50:07.988888 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.988683 4966 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-service-ca-bundle\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:07.989034 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.989013 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-trusted-ca-bundle" pod "341692ff-c7ac-4387-b9a3-d41ddd8f2ad3" (UID: "341692ff-c7ac-4387-b9a3-d41ddd8f2ad3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:50:07.990712 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.990689 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-kube-api-access-smztx" pod "09f9c9c6-4fae-443b-bd39-ce8b75bdb009" (UID: "09f9c9c6-4fae-443b-bd39-ce8b75bdb009"). InnerVolumeSpecName "kube-api-access-smztx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:50:07.990712 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:07.990699 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-kube-api-access-lsvdh" pod "341692ff-c7ac-4387-b9a3-d41ddd8f2ad3" (UID: "341692ff-c7ac-4387-b9a3-d41ddd8f2ad3"). InnerVolumeSpecName "kube-api-access-lsvdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:50:08.089982 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:08.089941 4966 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-service-ca-bundle\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:08.089982 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:08.089981 4966 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-archives-path\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:08.089982 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:08.089990 4966 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-archives-path\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:08.089982 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:08.089999 4966 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-trusted-ca-bundle\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:08.090258 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:08.090008 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsvdh\" (UniqueName: \"kubernetes.io/projected/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-kube-api-access-lsvdh\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:08.090258 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:08.090018 4966 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3-trusted-ca-bundle\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:08.090258 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:08.090027 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smztx\" (UniqueName: \"kubernetes.io/projected/09f9c9c6-4fae-443b-bd39-ce8b75bdb009-kube-api-access-smztx\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:08.714348 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:08.714322 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" Nov 04 03:50:08.714348 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:08.714334 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xwdp2-kwf4s" event={"ID":"341692ff-c7ac-4387-b9a3-d41ddd8f2ad3","Type":"ContainerDied","Data":"747fe1cc8b3517919d67735d1b24ebf8bc097f87f185660ed9f0c0bd64c45773"} Nov 04 03:50:08.714880 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:08.714393 4966 scope.go:122] "RemoveContainer" containerID="0d406fb7c7275db00231e338cea8ff883fb5db9637204678bfffa553cf628eee" Nov 04 03:50:08.715832 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:08.715803 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" event={"ID":"09f9c9c6-4fae-443b-bd39-ce8b75bdb009","Type":"ContainerDied","Data":"014fd36d2cb2ac0775cb8768f7a3c65efc9aac096eb928fdc7946d23c2eb6693"} Nov 04 03:50:08.715832 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:08.715825 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hg8zs-dxlbt" Nov 04 03:50:08.722637 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:08.722615 4966 scope.go:122] "RemoveContainer" containerID="0ce563f3cf9ff64f2130a79e1831baa3bb63bbdba255532dbb32311dd8719f24" Nov 04 03:50:08.733196 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:08.733175 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-hg8zs-dxlbt"] Nov 04 03:50:08.738144 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:08.738125 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-hg8zs-dxlbt"] Nov 04 03:50:08.750890 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:08.750871 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-xwdp2-kwf4s"] Nov 04 03:50:08.755247 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:08.755227 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-xwdp2-kwf4s"] Nov 04 03:50:10.339118 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:10.339086 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f9c9c6-4fae-443b-bd39-ce8b75bdb009" path="/var/lib/kubelet/pods/09f9c9c6-4fae-443b-bd39-ce8b75bdb009/volumes" Nov 04 03:50:10.339480 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:10.339458 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="341692ff-c7ac-4387-b9a3-d41ddd8f2ad3" path="/var/lib/kubelet/pods/341692ff-c7ac-4387-b9a3-d41ddd8f2ad3/volumes" Nov 04 03:50:22.646728 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:22.646658 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" podUID="3ce83f95-758c-4fd2-9678-441fe4309da7" containerName="registry" containerID="cri-o://f62d86c803b06a35a9e344215dc4ccd6a3d905008806464b6900beb649595c90" gracePeriod=30 Nov 04 03:50:22.776591 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:22.776563 4966 generic.go:356] "Generic (PLEG): container finished" podID="3ce83f95-758c-4fd2-9678-441fe4309da7" containerID="f62d86c803b06a35a9e344215dc4ccd6a3d905008806464b6900beb649595c90" exitCode=0 Nov 04 03:50:22.776704 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:22.776635 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" event={"ID":"3ce83f95-758c-4fd2-9678-441fe4309da7","Type":"ContainerDied","Data":"f62d86c803b06a35a9e344215dc4ccd6a3d905008806464b6900beb649595c90"} Nov 04 03:50:22.896301 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:22.896275 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:50:22.922357 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:22.922288 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-image-registry-private-configuration\") pod \"3ce83f95-758c-4fd2-9678-441fe4309da7\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " Nov 04 03:50:22.922357 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:22.922352 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-certificates\") pod \"3ce83f95-758c-4fd2-9678-441fe4309da7\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " Nov 04 03:50:22.922654 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:22.922389 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-installation-pull-secrets\") pod \"3ce83f95-758c-4fd2-9678-441fe4309da7\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " Nov 04 03:50:22.922654 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:22.922412 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-bound-sa-token\") pod \"3ce83f95-758c-4fd2-9678-441fe4309da7\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " Nov 04 03:50:22.922654 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:22.922434 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-trusted-ca\") pod \"3ce83f95-758c-4fd2-9678-441fe4309da7\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " Nov 04 03:50:22.922654 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:22.922468 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls\") pod \"3ce83f95-758c-4fd2-9678-441fe4309da7\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " Nov 04 03:50:22.922654 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:22.922513 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-kube-api-access-sqp55\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-kube-api-access-sqp55\") pod \"3ce83f95-758c-4fd2-9678-441fe4309da7\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " Nov 04 03:50:22.922954 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:22.922774 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-certificates" pod "3ce83f95-758c-4fd2-9678-441fe4309da7" (UID: "3ce83f95-758c-4fd2-9678-441fe4309da7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:50:22.923435 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:22.923372 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-trusted-ca" pod "3ce83f95-758c-4fd2-9678-441fe4309da7" (UID: "3ce83f95-758c-4fd2-9678-441fe4309da7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:50:22.925319 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:22.925275 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-image-registry-private-configuration" pod "3ce83f95-758c-4fd2-9678-441fe4309da7" (UID: "3ce83f95-758c-4fd2-9678-441fe4309da7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:50:22.925715 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:22.925669 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls" pod "3ce83f95-758c-4fd2-9678-441fe4309da7" (UID: "3ce83f95-758c-4fd2-9678-441fe4309da7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:50:22.925715 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:22.925699 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-kube-api-access-sqp55" pod "3ce83f95-758c-4fd2-9678-441fe4309da7" (UID: "3ce83f95-758c-4fd2-9678-441fe4309da7"). InnerVolumeSpecName "kube-api-access-sqp55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:50:22.925946 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:22.925893 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-installation-pull-secrets" pod "3ce83f95-758c-4fd2-9678-441fe4309da7" (UID: "3ce83f95-758c-4fd2-9678-441fe4309da7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:50:22.926143 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:22.926125 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-bound-sa-token" pod "3ce83f95-758c-4fd2-9678-441fe4309da7" (UID: "3ce83f95-758c-4fd2-9678-441fe4309da7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:50:23.023248 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:23.023217 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce83f95-758c-4fd2-9678-441fe4309da7-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ce83f95-758c-4fd2-9678-441fe4309da7-ca-trust-extracted\") pod \"3ce83f95-758c-4fd2-9678-441fe4309da7\" (UID: \"3ce83f95-758c-4fd2-9678-441fe4309da7\") " Nov 04 03:50:23.023388 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:23.023353 4966 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-image-registry-private-configuration\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:23.023388 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:23.023365 4966 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-certificates\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:23.023388 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:23.023374 4966 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ce83f95-758c-4fd2-9678-441fe4309da7-installation-pull-secrets\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:23.023388 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:23.023384 4966 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-bound-sa-token\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:23.023519 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:23.023394 4966 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ce83f95-758c-4fd2-9678-441fe4309da7-trusted-ca\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:23.023519 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:23.023402 4966 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-registry-tls\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:23.023519 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:23.023410 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqp55\" (UniqueName: \"kubernetes.io/projected/3ce83f95-758c-4fd2-9678-441fe4309da7-kube-api-access-sqp55\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:23.029172 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:23.029146 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce83f95-758c-4fd2-9678-441fe4309da7-ca-trust-extracted" pod "3ce83f95-758c-4fd2-9678-441fe4309da7" (UID: "3ce83f95-758c-4fd2-9678-441fe4309da7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:50:23.124271 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:23.124235 4966 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ce83f95-758c-4fd2-9678-441fe4309da7-ca-trust-extracted\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:50:23.782582 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:23.782548 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" event={"ID":"3ce83f95-758c-4fd2-9678-441fe4309da7","Type":"ContainerDied","Data":"28f134bc49d55cad184f3ba02c34f198502f96feaf1ffe84acdd01ab1f045274"} Nov 04 03:50:23.782582 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:23.782583 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8648b894c6-t5rf2" Nov 04 03:50:23.783048 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:23.782608 4966 scope.go:122] "RemoveContainer" containerID="f62d86c803b06a35a9e344215dc4ccd6a3d905008806464b6900beb649595c90" Nov 04 03:50:23.805888 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:23.805862 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8648b894c6-t5rf2"] Nov 04 03:50:23.809750 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:23.809729 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8648b894c6-t5rf2"] Nov 04 03:50:24.339542 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:24.339514 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ce83f95-758c-4fd2-9678-441fe4309da7" path="/var/lib/kubelet/pods/3ce83f95-758c-4fd2-9678-441fe4309da7/volumes" Nov 04 03:50:35.675913 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:50:35.675874 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:50:36.834253 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:36.834221 4966 generic.go:356] "Generic (PLEG): container finished" podID="d2052e7f-5ee8-4066-bb29-cd227fa0965b" containerID="88e5477e3067072f083da4dc4eacbb4b27e67337a6fd9221c9984b5e666c4f43" exitCode=0 Nov 04 03:50:36.834641 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:36.834273 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" event={"ID":"d2052e7f-5ee8-4066-bb29-cd227fa0965b","Type":"ContainerDied","Data":"88e5477e3067072f083da4dc4eacbb4b27e67337a6fd9221c9984b5e666c4f43"} Nov 04 03:50:36.834641 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:36.834562 4966 scope.go:122] "RemoveContainer" containerID="88e5477e3067072f083da4dc4eacbb4b27e67337a6fd9221c9984b5e666c4f43" Nov 04 03:50:37.840302 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:50:37.840268 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zdtm" event={"ID":"d2052e7f-5ee8-4066-bb29-cd227fa0965b","Type":"ContainerStarted","Data":"27ade1988b7c4360270dda15b05b12ae17019b708bddc08b0cce8d8f590d1d50"} Nov 04 03:51:05.676242 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:51:05.676205 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:51:35.675375 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:51:35.675288 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:52:05.676499 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:52:05.676461 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:52:35.676257 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:52:35.676219 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:52:56.766145 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.766113 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf"] Nov 04 03:52:56.766728 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.766587 4966 cpu_manager.go:414] "containerMap: removing container" podUID="3ce83f95-758c-4fd2-9678-441fe4309da7" containerName="registry" Nov 04 03:52:56.766728 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.766605 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce83f95-758c-4fd2-9678-441fe4309da7" containerName="registry" Nov 04 03:52:56.766728 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.766618 4966 cpu_manager.go:414] "containerMap: removing container" podUID="09f9c9c6-4fae-443b-bd39-ce8b75bdb009" containerName="insights-gathering" Nov 04 03:52:56.766728 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.766626 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f9c9c6-4fae-443b-bd39-ce8b75bdb009" containerName="insights-gathering" Nov 04 03:52:56.766728 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.766642 4966 cpu_manager.go:414] "containerMap: removing container" podUID="363d8742-c0a7-405c-9683-7b12afbd8bcc" containerName="registry" Nov 04 03:52:56.766728 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.766650 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363d8742-c0a7-405c-9683-7b12afbd8bcc" containerName="registry" Nov 04 03:52:56.766728 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.766664 4966 cpu_manager.go:414] "containerMap: removing container" podUID="341692ff-c7ac-4387-b9a3-d41ddd8f2ad3" containerName="insights-gathering" Nov 04 03:52:56.766728 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.766671 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="341692ff-c7ac-4387-b9a3-d41ddd8f2ad3" containerName="insights-gathering" Nov 04 03:52:56.767207 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.766757 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce83f95-758c-4fd2-9678-441fe4309da7" containerName="registry" Nov 04 03:52:56.767207 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.766770 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f9c9c6-4fae-443b-bd39-ce8b75bdb009" containerName="insights-gathering" Nov 04 03:52:56.767207 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.766779 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="363d8742-c0a7-405c-9683-7b12afbd8bcc" containerName="registry" Nov 04 03:52:56.767207 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.766788 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="341692ff-c7ac-4387-b9a3-d41ddd8f2ad3" containerName="insights-gathering" Nov 04 03:52:56.770005 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.769979 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.785461 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.785435 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf"] Nov 04 03:52:56.791411 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.791387 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-wmfnm\"" Nov 04 03:52:56.794706 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.794685 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 04 03:52:56.820793 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.820767 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12af8c07-513e-4b8c-8172-4aebb9731f66-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.820889 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.820796 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12af8c07-513e-4b8c-8172-4aebb9731f66-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.820889 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.820817 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12af8c07-513e-4b8c-8172-4aebb9731f66-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.820889 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.820840 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12af8c07-513e-4b8c-8172-4aebb9731f66-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.821029 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.820890 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvkt2\" (UniqueName: \"kubernetes.io/projected/12af8c07-513e-4b8c-8172-4aebb9731f66-kube-api-access-bvkt2\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.821068 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.821034 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12af8c07-513e-4b8c-8172-4aebb9731f66-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.821115 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.821092 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12af8c07-513e-4b8c-8172-4aebb9731f66-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.821165 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.821114 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12af8c07-513e-4b8c-8172-4aebb9731f66-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.821165 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.821132 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12af8c07-513e-4b8c-8172-4aebb9731f66-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.921764 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.921735 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12af8c07-513e-4b8c-8172-4aebb9731f66-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.921909 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.921766 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12af8c07-513e-4b8c-8172-4aebb9731f66-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.921909 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.921786 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12af8c07-513e-4b8c-8172-4aebb9731f66-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.921909 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.921805 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12af8c07-513e-4b8c-8172-4aebb9731f66-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.922094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.921958 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12af8c07-513e-4b8c-8172-4aebb9731f66-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.922094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.922006 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12af8c07-513e-4b8c-8172-4aebb9731f66-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.922094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.922039 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12af8c07-513e-4b8c-8172-4aebb9731f66-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.922094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.922068 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvkt2\" (UniqueName: \"kubernetes.io/projected/12af8c07-513e-4b8c-8172-4aebb9731f66-kube-api-access-bvkt2\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.922304 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.922131 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12af8c07-513e-4b8c-8172-4aebb9731f66-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.922304 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.922215 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12af8c07-513e-4b8c-8172-4aebb9731f66-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.922304 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.922237 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12af8c07-513e-4b8c-8172-4aebb9731f66-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.922304 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.922295 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12af8c07-513e-4b8c-8172-4aebb9731f66-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.922498 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.922378 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12af8c07-513e-4b8c-8172-4aebb9731f66-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.922847 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.922831 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12af8c07-513e-4b8c-8172-4aebb9731f66-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.924318 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.924297 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12af8c07-513e-4b8c-8172-4aebb9731f66-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.924578 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.924553 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12af8c07-513e-4b8c-8172-4aebb9731f66-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.934345 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.934322 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12af8c07-513e-4b8c-8172-4aebb9731f66-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:56.934451 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:56.934437 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvkt2\" (UniqueName: \"kubernetes.io/projected/12af8c07-513e-4b8c-8172-4aebb9731f66-kube-api-access-bvkt2\") pod \"openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf\" (UID: \"12af8c07-513e-4b8c-8172-4aebb9731f66\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:57.085645 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:57.085617 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:52:57.365686 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:57.365656 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf"] Nov 04 03:52:57.368044 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:52:57.368014 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12af8c07_513e_4b8c_8172_4aebb9731f66.slice/crio-4e016b176c318630ce15e85d7fd9d8882f9f22816ec3ad1da267369554334a51 WatchSource:0}: Error finding container 4e016b176c318630ce15e85d7fd9d8882f9f22816ec3ad1da267369554334a51: Status 404 returned error can't find the container with id 4e016b176c318630ce15e85d7fd9d8882f9f22816ec3ad1da267369554334a51 Nov 04 03:52:57.380161 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:57.380132 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" event={"ID":"12af8c07-513e-4b8c-8172-4aebb9731f66","Type":"ContainerStarted","Data":"4e016b176c318630ce15e85d7fd9d8882f9f22816ec3ad1da267369554334a51"} Nov 04 03:52:59.784944 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.784849 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk"] Nov 04 03:52:59.788215 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.788198 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk" Nov 04 03:52:59.793457 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.793432 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-gsbt4\"" Nov 04 03:52:59.793558 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.793512 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 04 03:52:59.793558 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.793513 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 04 03:52:59.799281 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.799260 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk"] Nov 04 03:52:59.818799 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.818764 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5"] Nov 04 03:52:59.822217 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.822195 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5" Nov 04 03:52:59.833886 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.833863 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5"] Nov 04 03:52:59.850526 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.850491 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e65383ce-37c4-4aa9-b745-6761add21bb7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk\" (UID: \"e65383ce-37c4-4aa9-b745-6761add21bb7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk" Nov 04 03:52:59.850641 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.850539 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsdlx\" (UniqueName: \"kubernetes.io/projected/e65383ce-37c4-4aa9-b745-6761add21bb7-kube-api-access-bsdlx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk\" (UID: \"e65383ce-37c4-4aa9-b745-6761add21bb7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk" Nov 04 03:52:59.850710 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.850644 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e65383ce-37c4-4aa9-b745-6761add21bb7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk\" (UID: \"e65383ce-37c4-4aa9-b745-6761add21bb7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk" Nov 04 03:52:59.851117 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.851097 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7"] Nov 04 03:52:59.854500 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.854482 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7" Nov 04 03:52:59.866872 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.866852 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7"] Nov 04 03:52:59.885414 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.885393 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p"] Nov 04 03:52:59.888661 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.888643 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p" Nov 04 03:52:59.892349 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.892324 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-h22ls\"" Nov 04 03:52:59.892434 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.892358 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-hspm7\"" Nov 04 03:52:59.892434 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.892418 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-2blc5\"" Nov 04 03:52:59.898831 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.898810 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p"] Nov 04 03:52:59.951228 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.951200 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7\" (UID: \"1d7dea63-fb3c-44ec-810c-cbd92f0ed448\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7" Nov 04 03:52:59.951356 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.951230 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/817b4162-3672-4690-a55e-b2684af29687-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p\" (UID: \"817b4162-3672-4690-a55e-b2684af29687\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p" Nov 04 03:52:59.951356 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.951248 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b72c4ff1-3427-4a9f-a236-6691591e3c98-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5\" (UID: \"b72c4ff1-3427-4a9f-a236-6691591e3c98\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5" Nov 04 03:52:59.951356 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.951269 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b72c4ff1-3427-4a9f-a236-6691591e3c98-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5\" (UID: \"b72c4ff1-3427-4a9f-a236-6691591e3c98\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5" Nov 04 03:52:59.951493 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.951409 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9ztx\" (UniqueName: \"kubernetes.io/projected/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-kube-api-access-c9ztx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7\" (UID: \"1d7dea63-fb3c-44ec-810c-cbd92f0ed448\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7" Nov 04 03:52:59.951493 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.951469 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e65383ce-37c4-4aa9-b745-6761add21bb7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk\" (UID: \"e65383ce-37c4-4aa9-b745-6761add21bb7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk" Nov 04 03:52:59.951586 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.951506 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/817b4162-3672-4690-a55e-b2684af29687-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p\" (UID: \"817b4162-3672-4690-a55e-b2684af29687\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p" Nov 04 03:52:59.951586 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.951525 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz5cg\" (UniqueName: \"kubernetes.io/projected/817b4162-3672-4690-a55e-b2684af29687-kube-api-access-rz5cg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p\" (UID: \"817b4162-3672-4690-a55e-b2684af29687\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p" Nov 04 03:52:59.951586 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.951561 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e65383ce-37c4-4aa9-b745-6761add21bb7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk\" (UID: \"e65383ce-37c4-4aa9-b745-6761add21bb7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk" Nov 04 03:52:59.951737 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.951598 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsdlx\" (UniqueName: \"kubernetes.io/projected/e65383ce-37c4-4aa9-b745-6761add21bb7-kube-api-access-bsdlx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk\" (UID: \"e65383ce-37c4-4aa9-b745-6761add21bb7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk" Nov 04 03:52:59.951737 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.951661 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6h7t\" (UniqueName: \"kubernetes.io/projected/b72c4ff1-3427-4a9f-a236-6691591e3c98-kube-api-access-v6h7t\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5\" (UID: \"b72c4ff1-3427-4a9f-a236-6691591e3c98\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5" Nov 04 03:52:59.951737 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.951694 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7\" (UID: \"1d7dea63-fb3c-44ec-810c-cbd92f0ed448\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7" Nov 04 03:52:59.952020 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.951998 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e65383ce-37c4-4aa9-b745-6761add21bb7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk\" (UID: \"e65383ce-37c4-4aa9-b745-6761add21bb7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk" Nov 04 03:52:59.952020 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.952009 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e65383ce-37c4-4aa9-b745-6761add21bb7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk\" (UID: \"e65383ce-37c4-4aa9-b745-6761add21bb7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk" Nov 04 03:52:59.964888 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:52:59.964863 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsdlx\" (UniqueName: \"kubernetes.io/projected/e65383ce-37c4-4aa9-b745-6761add21bb7-kube-api-access-bsdlx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk\" (UID: \"e65383ce-37c4-4aa9-b745-6761add21bb7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk" Nov 04 03:53:00.052796 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.052695 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7\" (UID: \"1d7dea63-fb3c-44ec-810c-cbd92f0ed448\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7" Nov 04 03:53:00.052796 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.052755 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7\" (UID: \"1d7dea63-fb3c-44ec-810c-cbd92f0ed448\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7" Nov 04 03:53:00.052796 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.052786 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/817b4162-3672-4690-a55e-b2684af29687-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p\" (UID: \"817b4162-3672-4690-a55e-b2684af29687\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p" Nov 04 03:53:00.053123 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.052811 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b72c4ff1-3427-4a9f-a236-6691591e3c98-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5\" (UID: \"b72c4ff1-3427-4a9f-a236-6691591e3c98\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5" Nov 04 03:53:00.053123 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.052843 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b72c4ff1-3427-4a9f-a236-6691591e3c98-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5\" (UID: \"b72c4ff1-3427-4a9f-a236-6691591e3c98\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5" Nov 04 03:53:00.053123 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.052878 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9ztx\" (UniqueName: \"kubernetes.io/projected/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-kube-api-access-c9ztx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7\" (UID: \"1d7dea63-fb3c-44ec-810c-cbd92f0ed448\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7" Nov 04 03:53:00.053123 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.052964 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/817b4162-3672-4690-a55e-b2684af29687-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p\" (UID: \"817b4162-3672-4690-a55e-b2684af29687\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p" Nov 04 03:53:00.053123 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.052993 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz5cg\" (UniqueName: \"kubernetes.io/projected/817b4162-3672-4690-a55e-b2684af29687-kube-api-access-rz5cg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p\" (UID: \"817b4162-3672-4690-a55e-b2684af29687\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p" Nov 04 03:53:00.053123 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.053055 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6h7t\" (UniqueName: \"kubernetes.io/projected/b72c4ff1-3427-4a9f-a236-6691591e3c98-kube-api-access-v6h7t\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5\" (UID: \"b72c4ff1-3427-4a9f-a236-6691591e3c98\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5" Nov 04 03:53:00.053419 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.053260 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b72c4ff1-3427-4a9f-a236-6691591e3c98-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5\" (UID: \"b72c4ff1-3427-4a9f-a236-6691591e3c98\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5" Nov 04 03:53:00.053419 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.053274 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/817b4162-3672-4690-a55e-b2684af29687-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p\" (UID: \"817b4162-3672-4690-a55e-b2684af29687\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p" Nov 04 03:53:00.053419 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.053280 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b72c4ff1-3427-4a9f-a236-6691591e3c98-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5\" (UID: \"b72c4ff1-3427-4a9f-a236-6691591e3c98\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5" Nov 04 03:53:00.053419 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.053294 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7\" (UID: \"1d7dea63-fb3c-44ec-810c-cbd92f0ed448\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7" Nov 04 03:53:00.053419 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.053260 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7\" (UID: \"1d7dea63-fb3c-44ec-810c-cbd92f0ed448\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7" Nov 04 03:53:00.053638 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.053516 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/817b4162-3672-4690-a55e-b2684af29687-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p\" (UID: \"817b4162-3672-4690-a55e-b2684af29687\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p" Nov 04 03:53:00.065845 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.065815 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz5cg\" (UniqueName: \"kubernetes.io/projected/817b4162-3672-4690-a55e-b2684af29687-kube-api-access-rz5cg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p\" (UID: \"817b4162-3672-4690-a55e-b2684af29687\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p" Nov 04 03:53:00.065985 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.065863 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9ztx\" (UniqueName: \"kubernetes.io/projected/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-kube-api-access-c9ztx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7\" (UID: \"1d7dea63-fb3c-44ec-810c-cbd92f0ed448\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7" Nov 04 03:53:00.066069 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.066047 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6h7t\" (UniqueName: \"kubernetes.io/projected/b72c4ff1-3427-4a9f-a236-6691591e3c98-kube-api-access-v6h7t\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5\" (UID: \"b72c4ff1-3427-4a9f-a236-6691591e3c98\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5" Nov 04 03:53:00.105335 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.105310 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk" Nov 04 03:53:00.137785 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.137748 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5" Nov 04 03:53:00.170418 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.170223 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7" Nov 04 03:53:00.205308 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.205276 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p" Nov 04 03:53:00.382430 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.382409 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk"] Nov 04 03:53:00.384552 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:53:00.384521 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65383ce_37c4_4aa9_b745_6761add21bb7.slice/crio-f8cf326ba8c79dc41b11cf1dfc254717fdb0786f506f5c50ba549602bc649744 WatchSource:0}: Error finding container f8cf326ba8c79dc41b11cf1dfc254717fdb0786f506f5c50ba549602bc649744: Status 404 returned error can't find the container with id f8cf326ba8c79dc41b11cf1dfc254717fdb0786f506f5c50ba549602bc649744 Nov 04 03:53:00.393972 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.393947 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk" event={"ID":"e65383ce-37c4-4aa9-b745-6761add21bb7","Type":"ContainerStarted","Data":"f8cf326ba8c79dc41b11cf1dfc254717fdb0786f506f5c50ba549602bc649744"} Nov 04 03:53:00.419135 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.419109 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5"] Nov 04 03:53:00.421546 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:53:00.421523 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72c4ff1_3427_4a9f_a236_6691591e3c98.slice/crio-71e1571f2864af976def6549b7374bce3f5490e5c6d8d24bda6ec071c6ed8928 WatchSource:0}: Error finding container 71e1571f2864af976def6549b7374bce3f5490e5c6d8d24bda6ec071c6ed8928: Status 404 returned error can't find the container with id 71e1571f2864af976def6549b7374bce3f5490e5c6d8d24bda6ec071c6ed8928 Nov 04 03:53:00.452382 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.452336 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7"] Nov 04 03:53:00.455750 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:53:00.455717 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7dea63_fb3c_44ec_810c_cbd92f0ed448.slice/crio-9f54465c63ea273c9d9dd0fc4b415eee597b1939dfcc1b9bc6f4f6a78f651caa WatchSource:0}: Error finding container 9f54465c63ea273c9d9dd0fc4b415eee597b1939dfcc1b9bc6f4f6a78f651caa: Status 404 returned error can't find the container with id 9f54465c63ea273c9d9dd0fc4b415eee597b1939dfcc1b9bc6f4f6a78f651caa Nov 04 03:53:00.489415 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:00.489387 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p"] Nov 04 03:53:01.402460 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:01.402381 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7" event={"ID":"1d7dea63-fb3c-44ec-810c-cbd92f0ed448","Type":"ContainerStarted","Data":"9f54465c63ea273c9d9dd0fc4b415eee597b1939dfcc1b9bc6f4f6a78f651caa"} Nov 04 03:53:01.404109 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:01.404035 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p" event={"ID":"817b4162-3672-4690-a55e-b2684af29687","Type":"ContainerStarted","Data":"4a36e85af645e77f6c191ebe07438d44bd998428c9ef8f032df42911a8d63c16"} Nov 04 03:53:01.405767 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:01.405740 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5" event={"ID":"b72c4ff1-3427-4a9f-a236-6691591e3c98","Type":"ContainerStarted","Data":"71e1571f2864af976def6549b7374bce3f5490e5c6d8d24bda6ec071c6ed8928"} Nov 04 03:53:05.726697 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:53:05.726654 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:53:07.137107 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:07.137071 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 03:53:07.137498 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:07.137142 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 03:53:07.137498 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:07.137173 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 03:53:07.453476 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:07.452659 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" event={"ID":"12af8c07-513e-4b8c-8172-4aebb9731f66","Type":"ContainerStarted","Data":"db73052b65f44c15b991a77a74de323c0799cb936164b52b5bfb7f9b919b3cba"} Nov 04 03:53:07.480620 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:07.480564 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" podStartSLOduration=1.714452227 podStartE2EDuration="11.480545781s" totalImagesPullingTime="9.766093554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:53:07.479008376 +0000 UTC m=+231.279954185" watchObservedRunningTime="2026-11-04 03:53:07.480545781 +0000 UTC m=+231.281491590" Nov 04 03:53:08.086757 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:08.086724 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:53:08.092633 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:08.092397 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:53:08.458830 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:08.458748 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:53:08.459969 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:08.459945 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf" Nov 04 03:53:10.468500 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:10.468462 4966 generic.go:356] "Generic (PLEG): container finished" podID="1d7dea63-fb3c-44ec-810c-cbd92f0ed448" containerID="ed82e0f9610def0304cf4fb59ee225a064f768cce3a5fd7c24cc6c0421260722" exitCode=0 Nov 04 03:53:10.468956 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:10.468547 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7" event={"ID":"1d7dea63-fb3c-44ec-810c-cbd92f0ed448","Type":"ContainerDied","Data":"ed82e0f9610def0304cf4fb59ee225a064f768cce3a5fd7c24cc6c0421260722"} Nov 04 03:53:10.470255 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:10.470224 4966 generic.go:356] "Generic (PLEG): container finished" podID="e65383ce-37c4-4aa9-b745-6761add21bb7" containerID="94ad6935bc436b4d4c3ab9eb1a4ad2491dac14e254c7ac0d9e8796db0f8d9206" exitCode=0 Nov 04 03:53:10.470410 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:10.470298 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk" event={"ID":"e65383ce-37c4-4aa9-b745-6761add21bb7","Type":"ContainerDied","Data":"94ad6935bc436b4d4c3ab9eb1a4ad2491dac14e254c7ac0d9e8796db0f8d9206"} Nov 04 03:53:10.472063 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:10.472035 4966 generic.go:356] "Generic (PLEG): container finished" podID="817b4162-3672-4690-a55e-b2684af29687" containerID="a1b36c29320cb0dfa1b6c9fa0ef6e4ed44baf82e644966208565fd75c49c6d74" exitCode=0 Nov 04 03:53:10.472135 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:10.472100 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p" event={"ID":"817b4162-3672-4690-a55e-b2684af29687","Type":"ContainerDied","Data":"a1b36c29320cb0dfa1b6c9fa0ef6e4ed44baf82e644966208565fd75c49c6d74"} Nov 04 03:53:10.473968 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:10.473949 4966 generic.go:356] "Generic (PLEG): container finished" podID="b72c4ff1-3427-4a9f-a236-6691591e3c98" containerID="4d980ad9155df95db42076f64a83ee7ff56595fae64698a23e4829f20938a69e" exitCode=0 Nov 04 03:53:10.474050 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:10.474030 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5" event={"ID":"b72c4ff1-3427-4a9f-a236-6691591e3c98","Type":"ContainerDied","Data":"4d980ad9155df95db42076f64a83ee7ff56595fae64698a23e4829f20938a69e"} Nov 04 03:53:11.479743 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:11.479717 4966 generic.go:356] "Generic (PLEG): container finished" podID="b72c4ff1-3427-4a9f-a236-6691591e3c98" containerID="1979bd880b2c8f20572a69240be3fae2a79ec4e1d77da9b77eb811567f88917c" exitCode=0 Nov 04 03:53:11.480155 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:11.479791 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5" event={"ID":"b72c4ff1-3427-4a9f-a236-6691591e3c98","Type":"ContainerDied","Data":"1979bd880b2c8f20572a69240be3fae2a79ec4e1d77da9b77eb811567f88917c"} Nov 04 03:53:11.481773 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:11.481746 4966 generic.go:356] "Generic (PLEG): container finished" podID="1d7dea63-fb3c-44ec-810c-cbd92f0ed448" containerID="8a9b425e8d51e518a709026cb6b293b21e9af61e5e49ac8c7d2bada1b6d05f7e" exitCode=0 Nov 04 03:53:11.481872 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:11.481802 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7" event={"ID":"1d7dea63-fb3c-44ec-810c-cbd92f0ed448","Type":"ContainerDied","Data":"8a9b425e8d51e518a709026cb6b293b21e9af61e5e49ac8c7d2bada1b6d05f7e"} Nov 04 03:53:12.490694 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:12.490653 4966 generic.go:356] "Generic (PLEG): container finished" podID="817b4162-3672-4690-a55e-b2684af29687" containerID="7794609d60393cda9da2d90b6b19c48e70f6e84ab65d53c2ce67c7e176d9310e" exitCode=0 Nov 04 03:53:12.491209 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:12.490766 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p" event={"ID":"817b4162-3672-4690-a55e-b2684af29687","Type":"ContainerDied","Data":"7794609d60393cda9da2d90b6b19c48e70f6e84ab65d53c2ce67c7e176d9310e"} Nov 04 03:53:13.499704 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:13.499661 4966 generic.go:356] "Generic (PLEG): container finished" podID="e65383ce-37c4-4aa9-b745-6761add21bb7" containerID="49cbd3cfc961ceabe2abf51d1ee8f3290b3b60e8498d7c67cef7325fabc049b0" exitCode=0 Nov 04 03:53:13.500212 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:13.499719 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk" event={"ID":"e65383ce-37c4-4aa9-b745-6761add21bb7","Type":"ContainerDied","Data":"49cbd3cfc961ceabe2abf51d1ee8f3290b3b60e8498d7c67cef7325fabc049b0"} Nov 04 03:53:21.548708 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:21.548448 4966 generic.go:356] "Generic (PLEG): container finished" podID="1d7dea63-fb3c-44ec-810c-cbd92f0ed448" containerID="2b368f9c6861749af38a3ebc3f75e7abaadace7148f1734172d0bc8f9e8f377d" exitCode=0 Nov 04 03:53:21.548708 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:21.548596 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7" event={"ID":"1d7dea63-fb3c-44ec-810c-cbd92f0ed448","Type":"ContainerDied","Data":"2b368f9c6861749af38a3ebc3f75e7abaadace7148f1734172d0bc8f9e8f377d"} Nov 04 03:53:21.552331 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:21.552295 4966 generic.go:356] "Generic (PLEG): container finished" podID="b72c4ff1-3427-4a9f-a236-6691591e3c98" containerID="be5c9c325bc3102eaa834e1f357520164ec8b09ea1f12ea3bbb85c391f7c8d3d" exitCode=0 Nov 04 03:53:21.552457 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:21.552382 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5" event={"ID":"b72c4ff1-3427-4a9f-a236-6691591e3c98","Type":"ContainerDied","Data":"be5c9c325bc3102eaa834e1f357520164ec8b09ea1f12ea3bbb85c391f7c8d3d"} Nov 04 03:53:22.558973 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.558938 4966 generic.go:356] "Generic (PLEG): container finished" podID="e65383ce-37c4-4aa9-b745-6761add21bb7" containerID="46ced6e4950ec394ff9c10773a708a346c011c29f00375df83b6a6d27b08ee8a" exitCode=0 Nov 04 03:53:22.559311 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.559014 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk" event={"ID":"e65383ce-37c4-4aa9-b745-6761add21bb7","Type":"ContainerDied","Data":"46ced6e4950ec394ff9c10773a708a346c011c29f00375df83b6a6d27b08ee8a"} Nov 04 03:53:22.561116 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.561093 4966 generic.go:356] "Generic (PLEG): container finished" podID="817b4162-3672-4690-a55e-b2684af29687" containerID="82ceefb65a82eb30ce21e72c66aee8281a276d641c21b620ac961629acc0446b" exitCode=0 Nov 04 03:53:22.561232 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.561169 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p" event={"ID":"817b4162-3672-4690-a55e-b2684af29687","Type":"ContainerDied","Data":"82ceefb65a82eb30ce21e72c66aee8281a276d641c21b620ac961629acc0446b"} Nov 04 03:53:22.685875 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.685851 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5" Nov 04 03:53:22.703742 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.703722 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7" Nov 04 03:53:22.781248 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.781222 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72c4ff1-3427-4a9f-a236-6691591e3c98-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b72c4ff1-3427-4a9f-a236-6691591e3c98-bundle\") pod \"b72c4ff1-3427-4a9f-a236-6691591e3c98\" (UID: \"b72c4ff1-3427-4a9f-a236-6691591e3c98\") " Nov 04 03:53:22.781396 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.781263 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72c4ff1-3427-4a9f-a236-6691591e3c98-kube-api-access-v6h7t\" (UniqueName: \"kubernetes.io/projected/b72c4ff1-3427-4a9f-a236-6691591e3c98-kube-api-access-v6h7t\") pod \"b72c4ff1-3427-4a9f-a236-6691591e3c98\" (UID: \"b72c4ff1-3427-4a9f-a236-6691591e3c98\") " Nov 04 03:53:22.781396 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.781330 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72c4ff1-3427-4a9f-a236-6691591e3c98-util\" (UniqueName: \"kubernetes.io/empty-dir/b72c4ff1-3427-4a9f-a236-6691591e3c98-util\") pod \"b72c4ff1-3427-4a9f-a236-6691591e3c98\" (UID: \"b72c4ff1-3427-4a9f-a236-6691591e3c98\") " Nov 04 03:53:22.781768 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.781744 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72c4ff1-3427-4a9f-a236-6691591e3c98-bundle" pod "b72c4ff1-3427-4a9f-a236-6691591e3c98" (UID: "b72c4ff1-3427-4a9f-a236-6691591e3c98"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:53:22.783457 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.783436 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72c4ff1-3427-4a9f-a236-6691591e3c98-kube-api-access-v6h7t" pod "b72c4ff1-3427-4a9f-a236-6691591e3c98" (UID: "b72c4ff1-3427-4a9f-a236-6691591e3c98"). InnerVolumeSpecName "kube-api-access-v6h7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:53:22.785974 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.785953 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72c4ff1-3427-4a9f-a236-6691591e3c98-util" pod "b72c4ff1-3427-4a9f-a236-6691591e3c98" (UID: "b72c4ff1-3427-4a9f-a236-6691591e3c98"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:53:22.882274 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.882235 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-bundle\") pod \"1d7dea63-fb3c-44ec-810c-cbd92f0ed448\" (UID: \"1d7dea63-fb3c-44ec-810c-cbd92f0ed448\") " Nov 04 03:53:22.882414 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.882355 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-util\" (UniqueName: \"kubernetes.io/empty-dir/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-util\") pod \"1d7dea63-fb3c-44ec-810c-cbd92f0ed448\" (UID: \"1d7dea63-fb3c-44ec-810c-cbd92f0ed448\") " Nov 04 03:53:22.882414 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.882382 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-kube-api-access-c9ztx\" (UniqueName: \"kubernetes.io/projected/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-kube-api-access-c9ztx\") pod \"1d7dea63-fb3c-44ec-810c-cbd92f0ed448\" (UID: \"1d7dea63-fb3c-44ec-810c-cbd92f0ed448\") " Nov 04 03:53:22.882533 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.882501 4966 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b72c4ff1-3427-4a9f-a236-6691591e3c98-bundle\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:53:22.882533 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.882512 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6h7t\" (UniqueName: \"kubernetes.io/projected/b72c4ff1-3427-4a9f-a236-6691591e3c98-kube-api-access-v6h7t\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:53:22.882533 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.882520 4966 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b72c4ff1-3427-4a9f-a236-6691591e3c98-util\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:53:22.882747 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.882722 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-bundle" pod "1d7dea63-fb3c-44ec-810c-cbd92f0ed448" (UID: "1d7dea63-fb3c-44ec-810c-cbd92f0ed448"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:53:22.884326 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.884306 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-kube-api-access-c9ztx" pod "1d7dea63-fb3c-44ec-810c-cbd92f0ed448" (UID: "1d7dea63-fb3c-44ec-810c-cbd92f0ed448"). InnerVolumeSpecName "kube-api-access-c9ztx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:53:22.887779 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.887759 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-util" pod "1d7dea63-fb3c-44ec-810c-cbd92f0ed448" (UID: "1d7dea63-fb3c-44ec-810c-cbd92f0ed448"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:53:22.983509 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.983449 4966 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-util\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:53:22.983509 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.983473 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9ztx\" (UniqueName: \"kubernetes.io/projected/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-kube-api-access-c9ztx\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:53:22.983509 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:22.983482 4966 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d7dea63-fb3c-44ec-810c-cbd92f0ed448-bundle\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:53:23.567737 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.567708 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5" Nov 04 03:53:23.568190 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.567730 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74rcn5" event={"ID":"b72c4ff1-3427-4a9f-a236-6691591e3c98","Type":"ContainerDied","Data":"71e1571f2864af976def6549b7374bce3f5490e5c6d8d24bda6ec071c6ed8928"} Nov 04 03:53:23.568190 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.567787 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71e1571f2864af976def6549b7374bce3f5490e5c6d8d24bda6ec071c6ed8928" Nov 04 03:53:23.569657 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.569636 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7" event={"ID":"1d7dea63-fb3c-44ec-810c-cbd92f0ed448","Type":"ContainerDied","Data":"9f54465c63ea273c9d9dd0fc4b415eee597b1939dfcc1b9bc6f4f6a78f651caa"} Nov 04 03:53:23.569760 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.569671 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f54465c63ea273c9d9dd0fc4b415eee597b1939dfcc1b9bc6f4f6a78f651caa" Nov 04 03:53:23.569760 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.569689 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qjlz7" Nov 04 03:53:23.706383 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.706359 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk" Nov 04 03:53:23.731794 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.731762 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p" Nov 04 03:53:23.889756 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.889680 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65383ce-37c4-4aa9-b745-6761add21bb7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e65383ce-37c4-4aa9-b745-6761add21bb7-bundle\") pod \"e65383ce-37c4-4aa9-b745-6761add21bb7\" (UID: \"e65383ce-37c4-4aa9-b745-6761add21bb7\") " Nov 04 03:53:23.889756 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.889734 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/817b4162-3672-4690-a55e-b2684af29687-util\" (UniqueName: \"kubernetes.io/empty-dir/817b4162-3672-4690-a55e-b2684af29687-util\") pod \"817b4162-3672-4690-a55e-b2684af29687\" (UID: \"817b4162-3672-4690-a55e-b2684af29687\") " Nov 04 03:53:23.889756 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.889757 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/817b4162-3672-4690-a55e-b2684af29687-kube-api-access-rz5cg\" (UniqueName: \"kubernetes.io/projected/817b4162-3672-4690-a55e-b2684af29687-kube-api-access-rz5cg\") pod \"817b4162-3672-4690-a55e-b2684af29687\" (UID: \"817b4162-3672-4690-a55e-b2684af29687\") " Nov 04 03:53:23.890065 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.889781 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65383ce-37c4-4aa9-b745-6761add21bb7-util\" (UniqueName: \"kubernetes.io/empty-dir/e65383ce-37c4-4aa9-b745-6761add21bb7-util\") pod \"e65383ce-37c4-4aa9-b745-6761add21bb7\" (UID: \"e65383ce-37c4-4aa9-b745-6761add21bb7\") " Nov 04 03:53:23.890065 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.889810 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65383ce-37c4-4aa9-b745-6761add21bb7-kube-api-access-bsdlx\" (UniqueName: \"kubernetes.io/projected/e65383ce-37c4-4aa9-b745-6761add21bb7-kube-api-access-bsdlx\") pod \"e65383ce-37c4-4aa9-b745-6761add21bb7\" (UID: \"e65383ce-37c4-4aa9-b745-6761add21bb7\") " Nov 04 03:53:23.890065 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.889841 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/817b4162-3672-4690-a55e-b2684af29687-bundle\" (UniqueName: \"kubernetes.io/empty-dir/817b4162-3672-4690-a55e-b2684af29687-bundle\") pod \"817b4162-3672-4690-a55e-b2684af29687\" (UID: \"817b4162-3672-4690-a55e-b2684af29687\") " Nov 04 03:53:23.890458 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.890423 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65383ce-37c4-4aa9-b745-6761add21bb7-bundle" pod "e65383ce-37c4-4aa9-b745-6761add21bb7" (UID: "e65383ce-37c4-4aa9-b745-6761add21bb7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:53:23.890590 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.890570 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/817b4162-3672-4690-a55e-b2684af29687-bundle" pod "817b4162-3672-4690-a55e-b2684af29687" (UID: "817b4162-3672-4690-a55e-b2684af29687"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:53:23.891982 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.891955 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65383ce-37c4-4aa9-b745-6761add21bb7-kube-api-access-bsdlx" pod "e65383ce-37c4-4aa9-b745-6761add21bb7" (UID: "e65383ce-37c4-4aa9-b745-6761add21bb7"). InnerVolumeSpecName "kube-api-access-bsdlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:53:23.892306 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.892283 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/817b4162-3672-4690-a55e-b2684af29687-kube-api-access-rz5cg" pod "817b4162-3672-4690-a55e-b2684af29687" (UID: "817b4162-3672-4690-a55e-b2684af29687"). InnerVolumeSpecName "kube-api-access-rz5cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:53:23.895352 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.895333 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/817b4162-3672-4690-a55e-b2684af29687-util" pod "817b4162-3672-4690-a55e-b2684af29687" (UID: "817b4162-3672-4690-a55e-b2684af29687"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:53:23.896218 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.896198 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65383ce-37c4-4aa9-b745-6761add21bb7-util" pod "e65383ce-37c4-4aa9-b745-6761add21bb7" (UID: "e65383ce-37c4-4aa9-b745-6761add21bb7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:53:23.991378 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.991337 4966 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e65383ce-37c4-4aa9-b745-6761add21bb7-bundle\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:53:23.991378 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.991372 4966 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/817b4162-3672-4690-a55e-b2684af29687-util\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:53:23.991378 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.991385 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz5cg\" (UniqueName: \"kubernetes.io/projected/817b4162-3672-4690-a55e-b2684af29687-kube-api-access-rz5cg\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:53:23.991616 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.991398 4966 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e65383ce-37c4-4aa9-b745-6761add21bb7-util\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:53:23.991616 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.991410 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsdlx\" (UniqueName: \"kubernetes.io/projected/e65383ce-37c4-4aa9-b745-6761add21bb7-kube-api-access-bsdlx\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:53:23.991616 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:23.991421 4966 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/817b4162-3672-4690-a55e-b2684af29687-bundle\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:53:24.575222 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:24.575190 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk" event={"ID":"e65383ce-37c4-4aa9-b745-6761add21bb7","Type":"ContainerDied","Data":"f8cf326ba8c79dc41b11cf1dfc254717fdb0786f506f5c50ba549602bc649744"} Nov 04 03:53:24.575676 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:24.575242 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8cf326ba8c79dc41b11cf1dfc254717fdb0786f506f5c50ba549602bc649744" Nov 04 03:53:24.575676 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:24.575202 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czn8dk" Nov 04 03:53:24.577148 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:24.577120 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p" event={"ID":"817b4162-3672-4690-a55e-b2684af29687","Type":"ContainerDied","Data":"4a36e85af645e77f6c191ebe07438d44bd998428c9ef8f032df42911a8d63c16"} Nov 04 03:53:24.577278 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:24.577164 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a36e85af645e77f6c191ebe07438d44bd998428c9ef8f032df42911a8d63c16" Nov 04 03:53:24.577278 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:24.577167 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7r95p" Nov 04 03:53:35.635298 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635136 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq"] Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635446 4966 cpu_manager.go:414] "containerMap: removing container" podUID="1d7dea63-fb3c-44ec-810c-cbd92f0ed448" containerName="util" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635456 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7dea63-fb3c-44ec-810c-cbd92f0ed448" containerName="util" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635470 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e65383ce-37c4-4aa9-b745-6761add21bb7" containerName="extract" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635475 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65383ce-37c4-4aa9-b745-6761add21bb7" containerName="extract" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635486 4966 cpu_manager.go:414] "containerMap: removing container" podUID="817b4162-3672-4690-a55e-b2684af29687" containerName="pull" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635491 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="817b4162-3672-4690-a55e-b2684af29687" containerName="pull" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635498 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e65383ce-37c4-4aa9-b745-6761add21bb7" containerName="pull" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635502 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65383ce-37c4-4aa9-b745-6761add21bb7" containerName="pull" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635510 4966 cpu_manager.go:414] "containerMap: removing container" podUID="1d7dea63-fb3c-44ec-810c-cbd92f0ed448" containerName="pull" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635515 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7dea63-fb3c-44ec-810c-cbd92f0ed448" containerName="pull" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635521 4966 cpu_manager.go:414] "containerMap: removing container" podUID="1d7dea63-fb3c-44ec-810c-cbd92f0ed448" containerName="extract" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635525 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7dea63-fb3c-44ec-810c-cbd92f0ed448" containerName="extract" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635532 4966 cpu_manager.go:414] "containerMap: removing container" podUID="817b4162-3672-4690-a55e-b2684af29687" containerName="extract" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635536 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="817b4162-3672-4690-a55e-b2684af29687" containerName="extract" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635549 4966 cpu_manager.go:414] "containerMap: removing container" podUID="b72c4ff1-3427-4a9f-a236-6691591e3c98" containerName="pull" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635554 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72c4ff1-3427-4a9f-a236-6691591e3c98" containerName="pull" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635563 4966 cpu_manager.go:414] "containerMap: removing container" podUID="b72c4ff1-3427-4a9f-a236-6691591e3c98" containerName="extract" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635567 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72c4ff1-3427-4a9f-a236-6691591e3c98" containerName="extract" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635577 4966 cpu_manager.go:414] "containerMap: removing container" podUID="b72c4ff1-3427-4a9f-a236-6691591e3c98" containerName="util" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635581 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72c4ff1-3427-4a9f-a236-6691591e3c98" containerName="util" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635588 4966 cpu_manager.go:414] "containerMap: removing container" podUID="817b4162-3672-4690-a55e-b2684af29687" containerName="util" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635592 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="817b4162-3672-4690-a55e-b2684af29687" containerName="util" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635599 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e65383ce-37c4-4aa9-b745-6761add21bb7" containerName="util" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635603 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65383ce-37c4-4aa9-b745-6761add21bb7" containerName="util" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635653 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="817b4162-3672-4690-a55e-b2684af29687" containerName="extract" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635663 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72c4ff1-3427-4a9f-a236-6691591e3c98" containerName="extract" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635670 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e65383ce-37c4-4aa9-b745-6761add21bb7" containerName="extract" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.635676 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d7dea63-fb3c-44ec-810c-cbd92f0ed448" containerName="extract" Nov 04 03:53:35.647126 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.638304 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:35.658293 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.658267 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 04 03:53:35.658557 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.658540 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 04 03:53:35.659539 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.659522 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 04 03:53:35.659954 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.659940 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-9csf6\"" Nov 04 03:53:35.660039 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.660025 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 04 03:53:35.660096 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.660030 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 04 03:53:35.675524 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.675497 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt8w9\" (UniqueName: \"kubernetes.io/projected/147eeb22-995f-45be-908b-8a6edce063b1-kube-api-access-mt8w9\") pod \"opendatahub-operator-controller-manager-74f9c8978b-zkjxq\" (UID: \"147eeb22-995f-45be-908b-8a6edce063b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:35.675678 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.675531 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/147eeb22-995f-45be-908b-8a6edce063b1-webhook-cert\") pod \"opendatahub-operator-controller-manager-74f9c8978b-zkjxq\" (UID: \"147eeb22-995f-45be-908b-8a6edce063b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:35.675678 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.675556 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/147eeb22-995f-45be-908b-8a6edce063b1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74f9c8978b-zkjxq\" (UID: \"147eeb22-995f-45be-908b-8a6edce063b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:35.675678 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.675574 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/147eeb22-995f-45be-908b-8a6edce063b1-manifests\") pod \"opendatahub-operator-controller-manager-74f9c8978b-zkjxq\" (UID: \"147eeb22-995f-45be-908b-8a6edce063b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:35.675845 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.675705 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/147eeb22-995f-45be-908b-8a6edce063b1-tmp\") pod \"opendatahub-operator-controller-manager-74f9c8978b-zkjxq\" (UID: \"147eeb22-995f-45be-908b-8a6edce063b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:35.675845 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.675746 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/147eeb22-995f-45be-908b-8a6edce063b1-metrics-cert\") pod \"opendatahub-operator-controller-manager-74f9c8978b-zkjxq\" (UID: \"147eeb22-995f-45be-908b-8a6edce063b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:35.676011 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:53:35.675987 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:53:35.718620 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.718588 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq"] Nov 04 03:53:35.776705 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.776680 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/147eeb22-995f-45be-908b-8a6edce063b1-tmp\") pod \"opendatahub-operator-controller-manager-74f9c8978b-zkjxq\" (UID: \"147eeb22-995f-45be-908b-8a6edce063b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:35.776839 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.776721 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/147eeb22-995f-45be-908b-8a6edce063b1-metrics-cert\") pod \"opendatahub-operator-controller-manager-74f9c8978b-zkjxq\" (UID: \"147eeb22-995f-45be-908b-8a6edce063b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:35.776839 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.776753 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt8w9\" (UniqueName: \"kubernetes.io/projected/147eeb22-995f-45be-908b-8a6edce063b1-kube-api-access-mt8w9\") pod \"opendatahub-operator-controller-manager-74f9c8978b-zkjxq\" (UID: \"147eeb22-995f-45be-908b-8a6edce063b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:35.776839 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.776783 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/147eeb22-995f-45be-908b-8a6edce063b1-webhook-cert\") pod \"opendatahub-operator-controller-manager-74f9c8978b-zkjxq\" (UID: \"147eeb22-995f-45be-908b-8a6edce063b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:35.776839 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.776821 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/147eeb22-995f-45be-908b-8a6edce063b1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74f9c8978b-zkjxq\" (UID: \"147eeb22-995f-45be-908b-8a6edce063b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:35.777071 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.776849 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/147eeb22-995f-45be-908b-8a6edce063b1-manifests\") pod \"opendatahub-operator-controller-manager-74f9c8978b-zkjxq\" (UID: \"147eeb22-995f-45be-908b-8a6edce063b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:35.777285 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.777250 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/147eeb22-995f-45be-908b-8a6edce063b1-manifests\") pod \"opendatahub-operator-controller-manager-74f9c8978b-zkjxq\" (UID: \"147eeb22-995f-45be-908b-8a6edce063b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:35.779380 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.779352 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/147eeb22-995f-45be-908b-8a6edce063b1-metrics-cert\") pod \"opendatahub-operator-controller-manager-74f9c8978b-zkjxq\" (UID: \"147eeb22-995f-45be-908b-8a6edce063b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:35.779462 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.779396 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/147eeb22-995f-45be-908b-8a6edce063b1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74f9c8978b-zkjxq\" (UID: \"147eeb22-995f-45be-908b-8a6edce063b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:35.779513 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.779504 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/147eeb22-995f-45be-908b-8a6edce063b1-tmp\") pod \"opendatahub-operator-controller-manager-74f9c8978b-zkjxq\" (UID: \"147eeb22-995f-45be-908b-8a6edce063b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:35.779675 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.779657 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/147eeb22-995f-45be-908b-8a6edce063b1-webhook-cert\") pod \"opendatahub-operator-controller-manager-74f9c8978b-zkjxq\" (UID: \"147eeb22-995f-45be-908b-8a6edce063b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:35.797589 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.797570 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt8w9\" (UniqueName: \"kubernetes.io/projected/147eeb22-995f-45be-908b-8a6edce063b1-kube-api-access-mt8w9\") pod \"opendatahub-operator-controller-manager-74f9c8978b-zkjxq\" (UID: \"147eeb22-995f-45be-908b-8a6edce063b1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:35.950903 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:35.950818 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:36.253856 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:36.253828 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq"] Nov 04 03:53:36.256741 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:53:36.256712 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147eeb22_995f_45be_908b_8a6edce063b1.slice/crio-2132844fe01c55c712f6f65826f8bb882af367624b544308496c681faffad53e WatchSource:0}: Error finding container 2132844fe01c55c712f6f65826f8bb882af367624b544308496c681faffad53e: Status 404 returned error can't find the container with id 2132844fe01c55c712f6f65826f8bb882af367624b544308496c681faffad53e Nov 04 03:53:36.630498 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:36.630463 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" event={"ID":"147eeb22-995f-45be-908b-8a6edce063b1","Type":"ContainerStarted","Data":"2132844fe01c55c712f6f65826f8bb882af367624b544308496c681faffad53e"} Nov 04 03:53:39.535264 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:39.535230 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-b659x"] Nov 04 03:53:39.538794 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:39.538774 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b659x" Nov 04 03:53:39.543835 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:39.543816 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:53:39.543956 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:39.543876 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 04 03:53:39.544013 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:39.543969 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-lsthc\"" Nov 04 03:53:39.548772 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:39.548753 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-b659x"] Nov 04 03:53:39.611288 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:39.611259 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3caf95e8-e1ce-488c-aa79-9d558ad1caed-tmp\") pod \"jobset-operator-674cd59dc5-b659x\" (UID: \"3caf95e8-e1ce-488c-aa79-9d558ad1caed\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b659x" Nov 04 03:53:39.611395 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:39.611309 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gtrf\" (UniqueName: \"kubernetes.io/projected/3caf95e8-e1ce-488c-aa79-9d558ad1caed-kube-api-access-2gtrf\") pod \"jobset-operator-674cd59dc5-b659x\" (UID: \"3caf95e8-e1ce-488c-aa79-9d558ad1caed\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b659x" Nov 04 03:53:39.646493 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:39.646465 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" event={"ID":"147eeb22-995f-45be-908b-8a6edce063b1","Type":"ContainerStarted","Data":"5dbc177c019b0d78158f7c4e0effefc350d7f9493b9a0492eda0b9e8ef2454db"} Nov 04 03:53:39.712292 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:39.712218 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3caf95e8-e1ce-488c-aa79-9d558ad1caed-tmp\") pod \"jobset-operator-674cd59dc5-b659x\" (UID: \"3caf95e8-e1ce-488c-aa79-9d558ad1caed\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b659x" Nov 04 03:53:39.712416 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:39.712301 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gtrf\" (UniqueName: \"kubernetes.io/projected/3caf95e8-e1ce-488c-aa79-9d558ad1caed-kube-api-access-2gtrf\") pod \"jobset-operator-674cd59dc5-b659x\" (UID: \"3caf95e8-e1ce-488c-aa79-9d558ad1caed\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b659x" Nov 04 03:53:39.712651 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:39.712630 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3caf95e8-e1ce-488c-aa79-9d558ad1caed-tmp\") pod \"jobset-operator-674cd59dc5-b659x\" (UID: \"3caf95e8-e1ce-488c-aa79-9d558ad1caed\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b659x" Nov 04 03:53:39.724535 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:39.724508 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gtrf\" (UniqueName: \"kubernetes.io/projected/3caf95e8-e1ce-488c-aa79-9d558ad1caed-kube-api-access-2gtrf\") pod \"jobset-operator-674cd59dc5-b659x\" (UID: \"3caf95e8-e1ce-488c-aa79-9d558ad1caed\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b659x" Nov 04 03:53:39.871270 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:39.871234 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b659x" Nov 04 03:53:40.235567 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:40.235517 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-b659x"] Nov 04 03:53:40.238289 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:53:40.238262 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3caf95e8_e1ce_488c_aa79_9d558ad1caed.slice/crio-b5147623e92376c5da80ec478b5e9f659d4908dc27435f2ffd4c5c080c72c4d4 WatchSource:0}: Error finding container b5147623e92376c5da80ec478b5e9f659d4908dc27435f2ffd4c5c080c72c4d4: Status 404 returned error can't find the container with id b5147623e92376c5da80ec478b5e9f659d4908dc27435f2ffd4c5c080c72c4d4 Nov 04 03:53:40.653246 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:40.653209 4966 generic.go:356] "Generic (PLEG): container finished" podID="147eeb22-995f-45be-908b-8a6edce063b1" containerID="5dbc177c019b0d78158f7c4e0effefc350d7f9493b9a0492eda0b9e8ef2454db" exitCode=0 Nov 04 03:53:40.653687 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:40.653290 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" event={"ID":"147eeb22-995f-45be-908b-8a6edce063b1","Type":"ContainerDied","Data":"5dbc177c019b0d78158f7c4e0effefc350d7f9493b9a0492eda0b9e8ef2454db"} Nov 04 03:53:40.654958 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:40.654749 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b659x" event={"ID":"3caf95e8-e1ce-488c-aa79-9d558ad1caed","Type":"ContainerStarted","Data":"b5147623e92376c5da80ec478b5e9f659d4908dc27435f2ffd4c5c080c72c4d4"} Nov 04 03:53:41.662547 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:41.662503 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" event={"ID":"147eeb22-995f-45be-908b-8a6edce063b1","Type":"ContainerStarted","Data":"d8412db5423bb4cadc14d7718c1477e7ee35129c87f762dd97a55751bcbeb7a3"} Nov 04 03:53:41.663040 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:41.662963 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:41.710164 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:41.709355 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" podStartSLOduration=2.674811281 podStartE2EDuration="6.709336344s" totalImagesPullingTime="3.536604447s" totalInitContainerRuntime="497.920616ms" isStatefulPod=false podCreationTimestamp="2026-11-04 03:53:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:53:41.70657478 +0000 UTC m=+265.507520591" watchObservedRunningTime="2026-11-04 03:53:41.709336344 +0000 UTC m=+265.510282154" Nov 04 03:53:44.687732 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:44.687689 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b659x" event={"ID":"3caf95e8-e1ce-488c-aa79-9d558ad1caed","Type":"ContainerStarted","Data":"f853d971c56bae28949b5cc3de8c63548a765a5bb94b7c5cfbaaec1d5a6c18a5"} Nov 04 03:53:44.708343 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:44.708293 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b659x" podStartSLOduration=2.011482401 podStartE2EDuration="5.708278757s" totalImagesPullingTime="3.696796356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:53:44.706692876 +0000 UTC m=+268.507638685" watchObservedRunningTime="2026-11-04 03:53:44.708278757 +0000 UTC m=+268.509224564" Nov 04 03:53:52.670355 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:52.670321 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-74f9c8978b-zkjxq" Nov 04 03:53:54.709612 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:54.709572 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb95x"] Nov 04 03:53:54.760768 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:54.760738 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb95x"] Nov 04 03:53:54.760956 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:54.760828 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb95x" Nov 04 03:53:54.764835 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:54.764810 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 04 03:53:54.765003 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:54.764985 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 04 03:53:54.766305 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:54.766287 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 04 03:53:54.766505 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:54.766486 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5d4vv\"" Nov 04 03:53:54.766580 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:54.766557 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 04 03:53:54.864244 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:54.864216 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2q7v\" (UniqueName: \"kubernetes.io/projected/5cb703ee-a590-4816-819d-2fb98480affa-kube-api-access-m2q7v\") pod \"kuadrant-console-plugin-66c4fb575d-vb95x\" (UID: \"5cb703ee-a590-4816-819d-2fb98480affa\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb95x" Nov 04 03:53:54.864404 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:54.864257 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5cb703ee-a590-4816-819d-2fb98480affa-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vb95x\" (UID: \"5cb703ee-a590-4816-819d-2fb98480affa\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb95x" Nov 04 03:53:54.864475 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:54.864398 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5cb703ee-a590-4816-819d-2fb98480affa-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vb95x\" (UID: \"5cb703ee-a590-4816-819d-2fb98480affa\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb95x" Nov 04 03:53:54.965170 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:54.965094 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5cb703ee-a590-4816-819d-2fb98480affa-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vb95x\" (UID: \"5cb703ee-a590-4816-819d-2fb98480affa\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb95x" Nov 04 03:53:54.965170 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:54.965133 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2q7v\" (UniqueName: \"kubernetes.io/projected/5cb703ee-a590-4816-819d-2fb98480affa-kube-api-access-m2q7v\") pod \"kuadrant-console-plugin-66c4fb575d-vb95x\" (UID: \"5cb703ee-a590-4816-819d-2fb98480affa\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb95x" Nov 04 03:53:54.965170 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:54.965162 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5cb703ee-a590-4816-819d-2fb98480affa-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vb95x\" (UID: \"5cb703ee-a590-4816-819d-2fb98480affa\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb95x" Nov 04 03:53:54.965745 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:54.965721 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5cb703ee-a590-4816-819d-2fb98480affa-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vb95x\" (UID: \"5cb703ee-a590-4816-819d-2fb98480affa\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb95x" Nov 04 03:53:54.967627 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:54.967603 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5cb703ee-a590-4816-819d-2fb98480affa-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vb95x\" (UID: \"5cb703ee-a590-4816-819d-2fb98480affa\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb95x" Nov 04 03:53:54.977847 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:54.977826 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2q7v\" (UniqueName: \"kubernetes.io/projected/5cb703ee-a590-4816-819d-2fb98480affa-kube-api-access-m2q7v\") pod \"kuadrant-console-plugin-66c4fb575d-vb95x\" (UID: \"5cb703ee-a590-4816-819d-2fb98480affa\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb95x" Nov 04 03:53:55.088817 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:55.088786 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb95x" Nov 04 03:53:55.417525 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:55.417498 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb95x"] Nov 04 03:53:55.419363 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:53:55.419334 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cb703ee_a590_4816_819d_2fb98480affa.slice/crio-8c342c3e8d39aa4a164df54d2070465d83337d9809b81dce420b39d349551292 WatchSource:0}: Error finding container 8c342c3e8d39aa4a164df54d2070465d83337d9809b81dce420b39d349551292: Status 404 returned error can't find the container with id 8c342c3e8d39aa4a164df54d2070465d83337d9809b81dce420b39d349551292 Nov 04 03:53:55.741209 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:55.741098 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb95x" event={"ID":"5cb703ee-a590-4816-819d-2fb98480affa","Type":"ContainerStarted","Data":"8c342c3e8d39aa4a164df54d2070465d83337d9809b81dce420b39d349551292"} Nov 04 03:53:56.736511 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:56.736478 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-nh8zl"] Nov 04 03:53:56.739146 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:56.739121 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nh8zl" Nov 04 03:53:56.750676 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:56.750459 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 04 03:53:56.750676 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:56.750558 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-rmtst\"" Nov 04 03:53:56.750676 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:56.750645 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 04 03:53:56.753094 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:56.753072 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-nh8zl"] Nov 04 03:53:56.880454 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:56.880414 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55b5bdc5-6b0a-4b3e-9c0c-3c0cfbb37a94-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-nh8zl\" (UID: \"55b5bdc5-6b0a-4b3e-9c0c-3c0cfbb37a94\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nh8zl" Nov 04 03:53:56.880630 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:56.880467 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c4dp\" (UniqueName: \"kubernetes.io/projected/55b5bdc5-6b0a-4b3e-9c0c-3c0cfbb37a94-kube-api-access-4c4dp\") pod \"cert-manager-webhook-5dbbbf75cc-nh8zl\" (UID: \"55b5bdc5-6b0a-4b3e-9c0c-3c0cfbb37a94\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nh8zl" Nov 04 03:53:56.981774 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:56.981724 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55b5bdc5-6b0a-4b3e-9c0c-3c0cfbb37a94-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-nh8zl\" (UID: \"55b5bdc5-6b0a-4b3e-9c0c-3c0cfbb37a94\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nh8zl" Nov 04 03:53:56.981983 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:56.981782 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c4dp\" (UniqueName: \"kubernetes.io/projected/55b5bdc5-6b0a-4b3e-9c0c-3c0cfbb37a94-kube-api-access-4c4dp\") pod \"cert-manager-webhook-5dbbbf75cc-nh8zl\" (UID: \"55b5bdc5-6b0a-4b3e-9c0c-3c0cfbb37a94\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nh8zl" Nov 04 03:53:57.004446 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:57.004373 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55b5bdc5-6b0a-4b3e-9c0c-3c0cfbb37a94-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-nh8zl\" (UID: \"55b5bdc5-6b0a-4b3e-9c0c-3c0cfbb37a94\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nh8zl" Nov 04 03:53:57.004640 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:57.004618 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c4dp\" (UniqueName: \"kubernetes.io/projected/55b5bdc5-6b0a-4b3e-9c0c-3c0cfbb37a94-kube-api-access-4c4dp\") pod \"cert-manager-webhook-5dbbbf75cc-nh8zl\" (UID: \"55b5bdc5-6b0a-4b3e-9c0c-3c0cfbb37a94\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nh8zl" Nov 04 03:53:57.059483 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:57.059452 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nh8zl" Nov 04 03:53:57.344175 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:57.344145 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-nh8zl"] Nov 04 03:53:57.346445 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:53:57.346412 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55b5bdc5_6b0a_4b3e_9c0c_3c0cfbb37a94.slice/crio-bbfeac8bf5dc9ed377dc20646c57e3707befd6970c2ef0f48b8690fa3e44439d WatchSource:0}: Error finding container bbfeac8bf5dc9ed377dc20646c57e3707befd6970c2ef0f48b8690fa3e44439d: Status 404 returned error can't find the container with id bbfeac8bf5dc9ed377dc20646c57e3707befd6970c2ef0f48b8690fa3e44439d Nov 04 03:53:57.756091 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:53:57.756000 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nh8zl" event={"ID":"55b5bdc5-6b0a-4b3e-9c0c-3c0cfbb37a94","Type":"ContainerStarted","Data":"bbfeac8bf5dc9ed377dc20646c57e3707befd6970c2ef0f48b8690fa3e44439d"} Nov 04 03:54:01.776841 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:01.776744 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb95x" event={"ID":"5cb703ee-a590-4816-819d-2fb98480affa","Type":"ContainerStarted","Data":"0d2f71044767b43611836cbef0f23047fbfdf3f20e5db380dde73561cb020f6b"} Nov 04 03:54:01.778446 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:01.778419 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nh8zl" event={"ID":"55b5bdc5-6b0a-4b3e-9c0c-3c0cfbb37a94","Type":"ContainerStarted","Data":"5bf90788957c5b1e8c0c1c1bc86a69f78624257594cd238ced851e337a4875bb"} Nov 04 03:54:01.778568 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:01.778467 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nh8zl" Nov 04 03:54:01.795521 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:01.795475 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb95x" podStartSLOduration=2.119701526 podStartE2EDuration="7.795462682s" totalImagesPullingTime="5.675761156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:54:01.795050441 +0000 UTC m=+285.595996250" watchObservedRunningTime="2026-11-04 03:54:01.795462682 +0000 UTC m=+285.596408489" Nov 04 03:54:01.817218 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:01.817170 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nh8zl" podStartSLOduration=1.789478559 podStartE2EDuration="5.81715621s" totalImagesPullingTime="4.027677651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:54:01.816288279 +0000 UTC m=+285.617234088" watchObservedRunningTime="2026-11-04 03:54:01.81715621 +0000 UTC m=+285.618102019" Nov 04 03:54:05.675563 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:54:05.675530 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:54:06.817125 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:06.817092 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-6scxx"] Nov 04 03:54:06.820622 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:06.820603 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-6scxx" Nov 04 03:54:06.833318 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:06.833293 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-xxg5r\"" Nov 04 03:54:06.834981 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:06.834956 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-6scxx"] Nov 04 03:54:06.975290 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:06.975257 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4cdr\" (UniqueName: \"kubernetes.io/projected/765315ac-4048-471c-baf5-94f34437bee1-kube-api-access-b4cdr\") pod \"cert-manager-597b95cb84-6scxx\" (UID: \"765315ac-4048-471c-baf5-94f34437bee1\") " pod="cert-manager/cert-manager-597b95cb84-6scxx" Nov 04 03:54:06.975459 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:06.975318 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/765315ac-4048-471c-baf5-94f34437bee1-bound-sa-token\") pod \"cert-manager-597b95cb84-6scxx\" (UID: \"765315ac-4048-471c-baf5-94f34437bee1\") " pod="cert-manager/cert-manager-597b95cb84-6scxx" Nov 04 03:54:07.075761 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:07.075688 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4cdr\" (UniqueName: \"kubernetes.io/projected/765315ac-4048-471c-baf5-94f34437bee1-kube-api-access-b4cdr\") pod \"cert-manager-597b95cb84-6scxx\" (UID: \"765315ac-4048-471c-baf5-94f34437bee1\") " pod="cert-manager/cert-manager-597b95cb84-6scxx" Nov 04 03:54:07.075761 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:07.075750 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/765315ac-4048-471c-baf5-94f34437bee1-bound-sa-token\") pod \"cert-manager-597b95cb84-6scxx\" (UID: \"765315ac-4048-471c-baf5-94f34437bee1\") " pod="cert-manager/cert-manager-597b95cb84-6scxx" Nov 04 03:54:07.090242 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:07.090206 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/765315ac-4048-471c-baf5-94f34437bee1-bound-sa-token\") pod \"cert-manager-597b95cb84-6scxx\" (UID: \"765315ac-4048-471c-baf5-94f34437bee1\") " pod="cert-manager/cert-manager-597b95cb84-6scxx" Nov 04 03:54:07.090242 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:07.090229 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4cdr\" (UniqueName: \"kubernetes.io/projected/765315ac-4048-471c-baf5-94f34437bee1-kube-api-access-b4cdr\") pod \"cert-manager-597b95cb84-6scxx\" (UID: \"765315ac-4048-471c-baf5-94f34437bee1\") " pod="cert-manager/cert-manager-597b95cb84-6scxx" Nov 04 03:54:07.132092 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:07.132065 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-6scxx" Nov 04 03:54:07.405562 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:07.405513 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-6scxx"] Nov 04 03:54:07.408172 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:54:07.408141 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod765315ac_4048_471c_baf5_94f34437bee1.slice/crio-b8a4a85325830dd4a45088e3e4f7b41e77b1118309c5ce212fb8ed20117141f0 WatchSource:0}: Error finding container b8a4a85325830dd4a45088e3e4f7b41e77b1118309c5ce212fb8ed20117141f0: Status 404 returned error can't find the container with id b8a4a85325830dd4a45088e3e4f7b41e77b1118309c5ce212fb8ed20117141f0 Nov 04 03:54:07.786237 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:07.786159 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nh8zl" Nov 04 03:54:07.808057 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:07.808024 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-6scxx" event={"ID":"765315ac-4048-471c-baf5-94f34437bee1","Type":"ContainerStarted","Data":"4ba6de8573ce8944f42b2ee637bc3a1bd3c4a9b4f9a4f0314b10a8a10198a6a8"} Nov 04 03:54:07.808057 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:07.808062 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-6scxx" event={"ID":"765315ac-4048-471c-baf5-94f34437bee1","Type":"ContainerStarted","Data":"b8a4a85325830dd4a45088e3e4f7b41e77b1118309c5ce212fb8ed20117141f0"} Nov 04 03:54:07.834988 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:07.834934 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-6scxx" podStartSLOduration=1.8349032269999999 podStartE2EDuration="1.834903227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:54:07.833312679 +0000 UTC m=+291.634258488" watchObservedRunningTime="2026-11-04 03:54:07.834903227 +0000 UTC m=+291.635849035" Nov 04 03:54:11.944212 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:11.944155 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5bd469f65f-x7t8v"] Nov 04 03:54:11.980277 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:11.980243 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bd469f65f-x7t8v"] Nov 04 03:54:11.980455 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:11.980346 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bd469f65f-x7t8v" Nov 04 03:54:11.989137 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:11.988999 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 04 03:54:11.990358 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:11.990331 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 04 03:54:11.990606 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:11.990586 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9wsr5\"" Nov 04 03:54:12.119123 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:12.119088 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcd98\" (UniqueName: \"kubernetes.io/projected/f376642e-e719-4112-858d-5dc9575b1236-kube-api-access-bcd98\") pod \"kube-auth-proxy-5bd469f65f-x7t8v\" (UID: \"f376642e-e719-4112-858d-5dc9575b1236\") " pod="openshift-ingress/kube-auth-proxy-5bd469f65f-x7t8v" Nov 04 03:54:12.119325 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:12.119174 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f376642e-e719-4112-858d-5dc9575b1236-tmp\") pod \"kube-auth-proxy-5bd469f65f-x7t8v\" (UID: \"f376642e-e719-4112-858d-5dc9575b1236\") " pod="openshift-ingress/kube-auth-proxy-5bd469f65f-x7t8v" Nov 04 03:54:12.119325 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:12.119207 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f376642e-e719-4112-858d-5dc9575b1236-tls-certs\") pod \"kube-auth-proxy-5bd469f65f-x7t8v\" (UID: \"f376642e-e719-4112-858d-5dc9575b1236\") " pod="openshift-ingress/kube-auth-proxy-5bd469f65f-x7t8v" Nov 04 03:54:12.220541 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:12.220455 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcd98\" (UniqueName: \"kubernetes.io/projected/f376642e-e719-4112-858d-5dc9575b1236-kube-api-access-bcd98\") pod \"kube-auth-proxy-5bd469f65f-x7t8v\" (UID: \"f376642e-e719-4112-858d-5dc9575b1236\") " pod="openshift-ingress/kube-auth-proxy-5bd469f65f-x7t8v" Nov 04 03:54:12.220541 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:12.220517 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f376642e-e719-4112-858d-5dc9575b1236-tmp\") pod \"kube-auth-proxy-5bd469f65f-x7t8v\" (UID: \"f376642e-e719-4112-858d-5dc9575b1236\") " pod="openshift-ingress/kube-auth-proxy-5bd469f65f-x7t8v" Nov 04 03:54:12.220791 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:12.220548 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f376642e-e719-4112-858d-5dc9575b1236-tls-certs\") pod \"kube-auth-proxy-5bd469f65f-x7t8v\" (UID: \"f376642e-e719-4112-858d-5dc9575b1236\") " pod="openshift-ingress/kube-auth-proxy-5bd469f65f-x7t8v" Nov 04 03:54:12.222903 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:12.222872 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f376642e-e719-4112-858d-5dc9575b1236-tmp\") pod \"kube-auth-proxy-5bd469f65f-x7t8v\" (UID: \"f376642e-e719-4112-858d-5dc9575b1236\") " pod="openshift-ingress/kube-auth-proxy-5bd469f65f-x7t8v" Nov 04 03:54:12.223257 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:12.223123 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f376642e-e719-4112-858d-5dc9575b1236-tls-certs\") pod \"kube-auth-proxy-5bd469f65f-x7t8v\" (UID: \"f376642e-e719-4112-858d-5dc9575b1236\") " pod="openshift-ingress/kube-auth-proxy-5bd469f65f-x7t8v" Nov 04 03:54:12.232867 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:12.232844 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcd98\" (UniqueName: \"kubernetes.io/projected/f376642e-e719-4112-858d-5dc9575b1236-kube-api-access-bcd98\") pod \"kube-auth-proxy-5bd469f65f-x7t8v\" (UID: \"f376642e-e719-4112-858d-5dc9575b1236\") " pod="openshift-ingress/kube-auth-proxy-5bd469f65f-x7t8v" Nov 04 03:54:12.292657 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:12.292626 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bd469f65f-x7t8v" Nov 04 03:54:12.576582 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:12.576551 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bd469f65f-x7t8v"] Nov 04 03:54:12.579465 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:54:12.579425 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf376642e_e719_4112_858d_5dc9575b1236.slice/crio-dc8a6108a3ed9276f88f9bd177c39dc6999bc884b004644b71b8d863d19eb281 WatchSource:0}: Error finding container dc8a6108a3ed9276f88f9bd177c39dc6999bc884b004644b71b8d863d19eb281: Status 404 returned error can't find the container with id dc8a6108a3ed9276f88f9bd177c39dc6999bc884b004644b71b8d863d19eb281 Nov 04 03:54:12.833710 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:12.833623 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bd469f65f-x7t8v" event={"ID":"f376642e-e719-4112-858d-5dc9575b1236","Type":"ContainerStarted","Data":"dc8a6108a3ed9276f88f9bd177c39dc6999bc884b004644b71b8d863d19eb281"} Nov 04 03:54:13.217524 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:13.217440 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-vc98n"] Nov 04 03:54:13.220876 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:13.220851 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-vc98n" Nov 04 03:54:13.228420 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:13.228394 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-vc98n"] Nov 04 03:54:13.313253 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:13.313224 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jxzfk\"" Nov 04 03:54:13.331611 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:13.331578 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdn65\" (UniqueName: \"kubernetes.io/projected/9c82f04c-238f-4ec5-b093-b8815e021056-kube-api-access-rdn65\") pod \"authorino-57f7f8659f-vc98n\" (UID: \"9c82f04c-238f-4ec5-b093-b8815e021056\") " pod="kuadrant-system/authorino-57f7f8659f-vc98n" Nov 04 03:54:13.433150 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:13.433112 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdn65\" (UniqueName: \"kubernetes.io/projected/9c82f04c-238f-4ec5-b093-b8815e021056-kube-api-access-rdn65\") pod \"authorino-57f7f8659f-vc98n\" (UID: \"9c82f04c-238f-4ec5-b093-b8815e021056\") " pod="kuadrant-system/authorino-57f7f8659f-vc98n" Nov 04 03:54:13.467247 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:13.467212 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdn65\" (UniqueName: \"kubernetes.io/projected/9c82f04c-238f-4ec5-b093-b8815e021056-kube-api-access-rdn65\") pod \"authorino-57f7f8659f-vc98n\" (UID: \"9c82f04c-238f-4ec5-b093-b8815e021056\") " pod="kuadrant-system/authorino-57f7f8659f-vc98n" Nov 04 03:54:13.541547 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:13.541443 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-vc98n" Nov 04 03:54:13.830898 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:13.830867 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-vc98n"] Nov 04 03:54:13.834399 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:54:13.834369 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c82f04c_238f_4ec5_b093_b8815e021056.slice/crio-c7b1bf4f25231e69e6ceec54133886c6b71bb6af0d69b040ceb491789fd17a52 WatchSource:0}: Error finding container c7b1bf4f25231e69e6ceec54133886c6b71bb6af0d69b040ceb491789fd17a52: Status 404 returned error can't find the container with id c7b1bf4f25231e69e6ceec54133886c6b71bb6af0d69b040ceb491789fd17a52 Nov 04 03:54:14.851229 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:14.851177 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-vc98n" event={"ID":"9c82f04c-238f-4ec5-b093-b8815e021056","Type":"ContainerStarted","Data":"c7b1bf4f25231e69e6ceec54133886c6b71bb6af0d69b040ceb491789fd17a52"} Nov 04 03:54:15.327572 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.326193 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5"] Nov 04 03:54:15.329760 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.329731 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5" Nov 04 03:54:15.335199 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.335093 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-6vtw2\"" Nov 04 03:54:15.337010 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.336962 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 04 03:54:15.337134 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.337007 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 04 03:54:15.340322 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.340292 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 04 03:54:15.342654 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.342634 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5"] Nov 04 03:54:15.451828 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.451501 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2f40e58-66ab-4754-bd77-ed8e3d20a818-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-sdgx5\" (UID: \"f2f40e58-66ab-4754-bd77-ed8e3d20a818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5" Nov 04 03:54:15.451828 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.451606 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2f40e58-66ab-4754-bd77-ed8e3d20a818-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-sdgx5\" (UID: \"f2f40e58-66ab-4754-bd77-ed8e3d20a818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5" Nov 04 03:54:15.451828 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.451630 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfpgw\" (UniqueName: \"kubernetes.io/projected/f2f40e58-66ab-4754-bd77-ed8e3d20a818-kube-api-access-mfpgw\") pod \"kserve-module-controller-manager-7566cf5fc8-sdgx5\" (UID: \"f2f40e58-66ab-4754-bd77-ed8e3d20a818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5" Nov 04 03:54:15.451828 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.451668 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f2f40e58-66ab-4754-bd77-ed8e3d20a818-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-sdgx5\" (UID: \"f2f40e58-66ab-4754-bd77-ed8e3d20a818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5" Nov 04 03:54:15.552977 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.552914 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2f40e58-66ab-4754-bd77-ed8e3d20a818-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-sdgx5\" (UID: \"f2f40e58-66ab-4754-bd77-ed8e3d20a818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5" Nov 04 03:54:15.553079 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.553034 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2f40e58-66ab-4754-bd77-ed8e3d20a818-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-sdgx5\" (UID: \"f2f40e58-66ab-4754-bd77-ed8e3d20a818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5" Nov 04 03:54:15.553079 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.553064 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfpgw\" (UniqueName: \"kubernetes.io/projected/f2f40e58-66ab-4754-bd77-ed8e3d20a818-kube-api-access-mfpgw\") pod \"kserve-module-controller-manager-7566cf5fc8-sdgx5\" (UID: \"f2f40e58-66ab-4754-bd77-ed8e3d20a818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5" Nov 04 03:54:15.553148 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.553103 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f2f40e58-66ab-4754-bd77-ed8e3d20a818-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-sdgx5\" (UID: \"f2f40e58-66ab-4754-bd77-ed8e3d20a818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5" Nov 04 03:54:15.553504 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.553472 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f2f40e58-66ab-4754-bd77-ed8e3d20a818-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-sdgx5\" (UID: \"f2f40e58-66ab-4754-bd77-ed8e3d20a818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5" Nov 04 03:54:15.554226 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.554184 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2f40e58-66ab-4754-bd77-ed8e3d20a818-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-sdgx5\" (UID: \"f2f40e58-66ab-4754-bd77-ed8e3d20a818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5" Nov 04 03:54:15.554642 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.554624 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2f40e58-66ab-4754-bd77-ed8e3d20a818-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-sdgx5\" (UID: \"f2f40e58-66ab-4754-bd77-ed8e3d20a818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5" Nov 04 03:54:15.568279 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.568255 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfpgw\" (UniqueName: \"kubernetes.io/projected/f2f40e58-66ab-4754-bd77-ed8e3d20a818-kube-api-access-mfpgw\") pod \"kserve-module-controller-manager-7566cf5fc8-sdgx5\" (UID: \"f2f40e58-66ab-4754-bd77-ed8e3d20a818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5" Nov 04 03:54:15.650286 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.650076 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5" Nov 04 03:54:15.863949 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.863836 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bd469f65f-x7t8v" event={"ID":"f376642e-e719-4112-858d-5dc9575b1236","Type":"ContainerStarted","Data":"a4ecb726a118da3e07d3a0deba28436b9826170a36194a8dfc46f29b7f82ab3b"} Nov 04 03:54:15.887644 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.887587 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5bd469f65f-x7t8v" podStartSLOduration=1.919656018 podStartE2EDuration="4.887568162s" totalImagesPullingTime="2.967912144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:54:15.885506619 +0000 UTC m=+299.686452427" watchObservedRunningTime="2026-11-04 03:54:15.887568162 +0000 UTC m=+299.688513972" Nov 04 03:54:15.951807 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:15.951768 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5"] Nov 04 03:54:16.368949 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:16.368898 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/3.log" Nov 04 03:54:16.369733 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:16.369711 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/3.log" Nov 04 03:54:16.872318 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:16.872277 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5" event={"ID":"f2f40e58-66ab-4754-bd77-ed8e3d20a818","Type":"ContainerStarted","Data":"e19cd46208136c57c15c5545a201a369cbe0bcb3d79bd78bcc9a37929b3901ae"} Nov 04 03:54:17.879515 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:17.879475 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-vc98n" event={"ID":"9c82f04c-238f-4ec5-b093-b8815e021056","Type":"ContainerStarted","Data":"9108f0782542c97afe11d31555f135a17e2bd3964f2ebfce24f1d2631470c6e5"} Nov 04 03:54:17.896664 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:17.896615 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-vc98n" podStartSLOduration=1.53739646 podStartE2EDuration="4.896602866s" totalImagesPullingTime="3.359206406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:54:17.89594137 +0000 UTC m=+301.696887171" watchObservedRunningTime="2026-11-04 03:54:17.896602866 +0000 UTC m=+301.697548681" Nov 04 03:54:18.886374 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:18.886338 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5" event={"ID":"f2f40e58-66ab-4754-bd77-ed8e3d20a818","Type":"ContainerStarted","Data":"7638ffcd4f2ec42d97523522e87c9b9d5d911e5da638e20684163782ad763c93"} Nov 04 03:54:18.886767 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:18.886641 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5" Nov 04 03:54:18.910496 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:18.910446 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5" podStartSLOduration=1.064182402 podStartE2EDuration="3.910432045s" totalImagesPullingTime="2.846249643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:54:18.909124549 +0000 UTC m=+302.710070358" watchObservedRunningTime="2026-11-04 03:54:18.910432045 +0000 UTC m=+302.711377883" Nov 04 03:54:28.507431 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:28.507358 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-vc98n"] Nov 04 03:54:28.507964 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:28.507542 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-vc98n" podUID="9c82f04c-238f-4ec5-b093-b8815e021056" containerName="authorino" containerID="cri-o://9108f0782542c97afe11d31555f135a17e2bd3964f2ebfce24f1d2631470c6e5" gracePeriod=30 Nov 04 03:54:28.757778 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:28.757721 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-vc98n" Nov 04 03:54:28.870218 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:28.870184 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c82f04c-238f-4ec5-b093-b8815e021056-kube-api-access-rdn65\" (UniqueName: \"kubernetes.io/projected/9c82f04c-238f-4ec5-b093-b8815e021056-kube-api-access-rdn65\") pod \"9c82f04c-238f-4ec5-b093-b8815e021056\" (UID: \"9c82f04c-238f-4ec5-b093-b8815e021056\") " Nov 04 03:54:28.872362 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:28.872331 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c82f04c-238f-4ec5-b093-b8815e021056-kube-api-access-rdn65" pod "9c82f04c-238f-4ec5-b093-b8815e021056" (UID: "9c82f04c-238f-4ec5-b093-b8815e021056"). InnerVolumeSpecName "kube-api-access-rdn65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:54:28.933272 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:28.933241 4966 generic.go:356] "Generic (PLEG): container finished" podID="9c82f04c-238f-4ec5-b093-b8815e021056" containerID="9108f0782542c97afe11d31555f135a17e2bd3964f2ebfce24f1d2631470c6e5" exitCode=0 Nov 04 03:54:28.933428 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:28.933285 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-vc98n" Nov 04 03:54:28.933428 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:28.933326 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-vc98n" event={"ID":"9c82f04c-238f-4ec5-b093-b8815e021056","Type":"ContainerDied","Data":"9108f0782542c97afe11d31555f135a17e2bd3964f2ebfce24f1d2631470c6e5"} Nov 04 03:54:28.933428 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:28.933381 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-vc98n" event={"ID":"9c82f04c-238f-4ec5-b093-b8815e021056","Type":"ContainerDied","Data":"c7b1bf4f25231e69e6ceec54133886c6b71bb6af0d69b040ceb491789fd17a52"} Nov 04 03:54:28.933428 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:28.933403 4966 scope.go:122] "RemoveContainer" containerID="9108f0782542c97afe11d31555f135a17e2bd3964f2ebfce24f1d2631470c6e5" Nov 04 03:54:28.942992 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:28.942902 4966 scope.go:122] "RemoveContainer" containerID="9108f0782542c97afe11d31555f135a17e2bd3964f2ebfce24f1d2631470c6e5" Nov 04 03:54:28.943222 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:54:28.943204 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9108f0782542c97afe11d31555f135a17e2bd3964f2ebfce24f1d2631470c6e5\": container with ID starting with 9108f0782542c97afe11d31555f135a17e2bd3964f2ebfce24f1d2631470c6e5 not found: ID does not exist" containerID="9108f0782542c97afe11d31555f135a17e2bd3964f2ebfce24f1d2631470c6e5" Nov 04 03:54:28.943278 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:28.943231 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9108f0782542c97afe11d31555f135a17e2bd3964f2ebfce24f1d2631470c6e5"} err="failed to get container status \"9108f0782542c97afe11d31555f135a17e2bd3964f2ebfce24f1d2631470c6e5\": rpc error: code = NotFound desc = could not find container \"9108f0782542c97afe11d31555f135a17e2bd3964f2ebfce24f1d2631470c6e5\": container with ID starting with 9108f0782542c97afe11d31555f135a17e2bd3964f2ebfce24f1d2631470c6e5 not found: ID does not exist" Nov 04 03:54:28.956216 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:28.956195 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-vc98n"] Nov 04 03:54:28.960162 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:28.960140 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-vc98n"] Nov 04 03:54:28.970860 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:28.970840 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdn65\" (UniqueName: \"kubernetes.io/projected/9c82f04c-238f-4ec5-b093-b8815e021056-kube-api-access-rdn65\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:54:30.340054 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:30.340020 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c82f04c-238f-4ec5-b093-b8815e021056" path="/var/lib/kubelet/pods/9c82f04c-238f-4ec5-b093-b8815e021056/volumes" Nov 04 03:54:35.679078 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:54:35.679041 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:54:39.892356 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:54:39.892325 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sdgx5" Nov 04 03:55:00.691851 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:00.691814 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8b95c9c66-xqc54"] Nov 04 03:55:00.692328 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:00.692175 4966 cpu_manager.go:414] "containerMap: removing container" podUID="9c82f04c-238f-4ec5-b093-b8815e021056" containerName="authorino" Nov 04 03:55:00.692328 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:00.692187 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c82f04c-238f-4ec5-b093-b8815e021056" containerName="authorino" Nov 04 03:55:00.692328 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:00.692262 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c82f04c-238f-4ec5-b093-b8815e021056" containerName="authorino" Nov 04 03:55:00.695149 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:00.695131 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8b95c9c66-xqc54" Nov 04 03:55:00.703778 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:00.703758 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vcph9\"" Nov 04 03:55:00.703903 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:00.703829 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 04 03:55:00.703903 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:00.703840 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 04 03:55:00.703903 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:00.703895 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 04 03:55:00.711744 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:00.711721 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-xqc54"] Nov 04 03:55:00.839723 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:00.839683 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47f40d1b-c510-4696-bc31-0c792fd06de4-metrics-certs\") pod \"maas-controller-8b95c9c66-xqc54\" (UID: \"47f40d1b-c510-4696-bc31-0c792fd06de4\") " pod="opendatahub/maas-controller-8b95c9c66-xqc54" Nov 04 03:55:00.839899 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:00.839760 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctn4r\" (UniqueName: \"kubernetes.io/projected/47f40d1b-c510-4696-bc31-0c792fd06de4-kube-api-access-ctn4r\") pod \"maas-controller-8b95c9c66-xqc54\" (UID: \"47f40d1b-c510-4696-bc31-0c792fd06de4\") " pod="opendatahub/maas-controller-8b95c9c66-xqc54" Nov 04 03:55:00.839899 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:00.839808 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47f40d1b-c510-4696-bc31-0c792fd06de4-webhook-cert\") pod \"maas-controller-8b95c9c66-xqc54\" (UID: \"47f40d1b-c510-4696-bc31-0c792fd06de4\") " pod="opendatahub/maas-controller-8b95c9c66-xqc54" Nov 04 03:55:00.940542 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:00.940507 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47f40d1b-c510-4696-bc31-0c792fd06de4-metrics-certs\") pod \"maas-controller-8b95c9c66-xqc54\" (UID: \"47f40d1b-c510-4696-bc31-0c792fd06de4\") " pod="opendatahub/maas-controller-8b95c9c66-xqc54" Nov 04 03:55:00.940725 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:00.940568 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctn4r\" (UniqueName: \"kubernetes.io/projected/47f40d1b-c510-4696-bc31-0c792fd06de4-kube-api-access-ctn4r\") pod \"maas-controller-8b95c9c66-xqc54\" (UID: \"47f40d1b-c510-4696-bc31-0c792fd06de4\") " pod="opendatahub/maas-controller-8b95c9c66-xqc54" Nov 04 03:55:00.940725 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:00.940588 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47f40d1b-c510-4696-bc31-0c792fd06de4-webhook-cert\") pod \"maas-controller-8b95c9c66-xqc54\" (UID: \"47f40d1b-c510-4696-bc31-0c792fd06de4\") " pod="opendatahub/maas-controller-8b95c9c66-xqc54" Nov 04 03:55:00.943165 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:00.943108 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47f40d1b-c510-4696-bc31-0c792fd06de4-metrics-certs\") pod \"maas-controller-8b95c9c66-xqc54\" (UID: \"47f40d1b-c510-4696-bc31-0c792fd06de4\") " pod="opendatahub/maas-controller-8b95c9c66-xqc54" Nov 04 03:55:00.943260 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:00.943211 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47f40d1b-c510-4696-bc31-0c792fd06de4-webhook-cert\") pod \"maas-controller-8b95c9c66-xqc54\" (UID: \"47f40d1b-c510-4696-bc31-0c792fd06de4\") " pod="opendatahub/maas-controller-8b95c9c66-xqc54" Nov 04 03:55:00.953742 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:00.953714 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctn4r\" (UniqueName: \"kubernetes.io/projected/47f40d1b-c510-4696-bc31-0c792fd06de4-kube-api-access-ctn4r\") pod \"maas-controller-8b95c9c66-xqc54\" (UID: \"47f40d1b-c510-4696-bc31-0c792fd06de4\") " pod="opendatahub/maas-controller-8b95c9c66-xqc54" Nov 04 03:55:01.008157 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:01.008132 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8b95c9c66-xqc54" Nov 04 03:55:01.039476 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:01.039446 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55cc4d9668-f6wtl"] Nov 04 03:55:01.043814 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:01.043789 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55cc4d9668-f6wtl" Nov 04 03:55:01.055773 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:01.055703 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55cc4d9668-f6wtl"] Nov 04 03:55:01.142297 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:01.142264 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59-metrics-certs\") pod \"maas-controller-55cc4d9668-f6wtl\" (UID: \"c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59\") " pod="opendatahub/maas-controller-55cc4d9668-f6wtl" Nov 04 03:55:01.142431 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:01.142304 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59-webhook-cert\") pod \"maas-controller-55cc4d9668-f6wtl\" (UID: \"c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59\") " pod="opendatahub/maas-controller-55cc4d9668-f6wtl" Nov 04 03:55:01.142471 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:01.142420 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rklpd\" (UniqueName: \"kubernetes.io/projected/c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59-kube-api-access-rklpd\") pod \"maas-controller-55cc4d9668-f6wtl\" (UID: \"c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59\") " pod="opendatahub/maas-controller-55cc4d9668-f6wtl" Nov 04 03:55:01.243277 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:01.243203 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59-metrics-certs\") pod \"maas-controller-55cc4d9668-f6wtl\" (UID: \"c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59\") " pod="opendatahub/maas-controller-55cc4d9668-f6wtl" Nov 04 03:55:01.243277 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:01.243245 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59-webhook-cert\") pod \"maas-controller-55cc4d9668-f6wtl\" (UID: \"c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59\") " pod="opendatahub/maas-controller-55cc4d9668-f6wtl" Nov 04 03:55:01.243447 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:01.243311 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rklpd\" (UniqueName: \"kubernetes.io/projected/c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59-kube-api-access-rklpd\") pod \"maas-controller-55cc4d9668-f6wtl\" (UID: \"c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59\") " pod="opendatahub/maas-controller-55cc4d9668-f6wtl" Nov 04 03:55:01.245751 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:01.245724 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59-metrics-certs\") pod \"maas-controller-55cc4d9668-f6wtl\" (UID: \"c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59\") " pod="opendatahub/maas-controller-55cc4d9668-f6wtl" Nov 04 03:55:01.245851 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:01.245759 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59-webhook-cert\") pod \"maas-controller-55cc4d9668-f6wtl\" (UID: \"c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59\") " pod="opendatahub/maas-controller-55cc4d9668-f6wtl" Nov 04 03:55:01.257840 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:01.257814 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rklpd\" (UniqueName: \"kubernetes.io/projected/c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59-kube-api-access-rklpd\") pod \"maas-controller-55cc4d9668-f6wtl\" (UID: \"c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59\") " pod="opendatahub/maas-controller-55cc4d9668-f6wtl" Nov 04 03:55:01.291360 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:01.291338 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-xqc54"] Nov 04 03:55:01.293526 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:55:01.293502 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f40d1b_c510_4696_bc31_0c792fd06de4.slice/crio-523d8e45273420504b6dd707756424333bc39d05562b2149b8160090753af6c7 WatchSource:0}: Error finding container 523d8e45273420504b6dd707756424333bc39d05562b2149b8160090753af6c7: Status 404 returned error can't find the container with id 523d8e45273420504b6dd707756424333bc39d05562b2149b8160090753af6c7 Nov 04 03:55:01.294989 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:01.294973 4966 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 03:55:01.359732 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:01.359701 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55cc4d9668-f6wtl" Nov 04 03:55:01.641266 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:01.641237 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55cc4d9668-f6wtl"] Nov 04 03:55:01.643099 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:55:01.643075 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c6cd4f_a5bc_4602_b6cc_e8ed6595ed59.slice/crio-2e2233ce039e929bcc5ca678668d6e430d73f0319f1cf56405f2c1a3dfca18e1 WatchSource:0}: Error finding container 2e2233ce039e929bcc5ca678668d6e430d73f0319f1cf56405f2c1a3dfca18e1: Status 404 returned error can't find the container with id 2e2233ce039e929bcc5ca678668d6e430d73f0319f1cf56405f2c1a3dfca18e1 Nov 04 03:55:02.090692 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:02.090654 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8b95c9c66-xqc54" event={"ID":"47f40d1b-c510-4696-bc31-0c792fd06de4","Type":"ContainerStarted","Data":"523d8e45273420504b6dd707756424333bc39d05562b2149b8160090753af6c7"} Nov 04 03:55:02.091861 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:02.091838 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55cc4d9668-f6wtl" event={"ID":"c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59","Type":"ContainerStarted","Data":"2e2233ce039e929bcc5ca678668d6e430d73f0319f1cf56405f2c1a3dfca18e1"} Nov 04 03:55:05.114146 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:05.114084 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55cc4d9668-f6wtl" event={"ID":"c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59","Type":"ContainerStarted","Data":"991ba668a2aca47ff5cac314237d6ff142fd8c05ee1e52cfbb2b4f4ecf2d8e21"} Nov 04 03:55:05.114616 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:05.114161 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55cc4d9668-f6wtl" Nov 04 03:55:05.149105 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:05.149050 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-55cc4d9668-f6wtl" podStartSLOduration=1.027166351 podStartE2EDuration="4.149036918s" totalImagesPullingTime="3.121870567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:55:05.148080991 +0000 UTC m=+348.949026801" watchObservedRunningTime="2026-11-04 03:55:05.149036918 +0000 UTC m=+348.949982726" Nov 04 03:55:05.677186 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:55:05.677142 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:55:06.119942 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:06.119887 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8b95c9c66-xqc54" event={"ID":"47f40d1b-c510-4696-bc31-0c792fd06de4","Type":"ContainerStarted","Data":"5cda23fc210b013990ff245fa57963fa520090e16e3d984028ce890868af5245"} Nov 04 03:55:06.120368 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:06.120123 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8b95c9c66-xqc54" Nov 04 03:55:06.161077 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:06.161029 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8b95c9c66-xqc54" podStartSLOduration=2.184034282 podStartE2EDuration="6.161014917s" totalImagesPullingTime="3.976980635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:55:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:55:06.159162579 +0000 UTC m=+349.960108386" watchObservedRunningTime="2026-11-04 03:55:06.161014917 +0000 UTC m=+349.961960725" Nov 04 03:55:10.871803 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:10.871763 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-58vj8"] Nov 04 03:55:10.879258 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:10.879234 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-58vj8" Nov 04 03:55:10.893051 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:10.886653 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-58vj8"] Nov 04 03:55:10.896671 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:10.896645 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 04 03:55:10.896808 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:10.896782 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8b8mb\"" Nov 04 03:55:10.917511 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:10.917477 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-2m46k"] Nov 04 03:55:10.921899 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:10.921876 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-2m46k" Nov 04 03:55:10.941314 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:10.941291 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 04 03:55:10.941776 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:10.941747 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-d5j9c\"" Nov 04 03:55:10.942413 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:10.942393 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-2m46k"] Nov 04 03:55:10.944959 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:10.944933 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-xdp9r"] Nov 04 03:55:10.949129 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:10.949108 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-xdp9r" Nov 04 03:55:10.972118 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:10.959399 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 04 03:55:10.972118 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:10.959473 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-8pdqn\"" Nov 04 03:55:10.972118 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:10.959583 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 04 03:55:10.972118 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:10.964171 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-xdp9r"] Nov 04 03:55:11.042282 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.042248 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwc8x\" (UniqueName: \"kubernetes.io/projected/545fb734-bf40-43f8-a19d-0fe82840ef71-kube-api-access-xwc8x\") pod \"kserve-controller-manager-776cc66d47-58vj8\" (UID: \"545fb734-bf40-43f8-a19d-0fe82840ef71\") " pod="opendatahub/kserve-controller-manager-776cc66d47-58vj8" Nov 04 03:55:11.042444 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.042286 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bffcd9a5-5579-4223-ad60-f1c5a862645f-tls-certs\") pod \"model-serving-api-75bd89fdfc-2m46k\" (UID: \"bffcd9a5-5579-4223-ad60-f1c5a862645f\") " pod="opendatahub/model-serving-api-75bd89fdfc-2m46k" Nov 04 03:55:11.042496 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.042447 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/545fb734-bf40-43f8-a19d-0fe82840ef71-cert\") pod \"kserve-controller-manager-776cc66d47-58vj8\" (UID: \"545fb734-bf40-43f8-a19d-0fe82840ef71\") " pod="opendatahub/kserve-controller-manager-776cc66d47-58vj8" Nov 04 03:55:11.042496 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.042480 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhpz4\" (UniqueName: \"kubernetes.io/projected/bffcd9a5-5579-4223-ad60-f1c5a862645f-kube-api-access-bhpz4\") pod \"model-serving-api-75bd89fdfc-2m46k\" (UID: \"bffcd9a5-5579-4223-ad60-f1c5a862645f\") " pod="opendatahub/model-serving-api-75bd89fdfc-2m46k" Nov 04 03:55:11.143119 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.143039 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwc8x\" (UniqueName: \"kubernetes.io/projected/545fb734-bf40-43f8-a19d-0fe82840ef71-kube-api-access-xwc8x\") pod \"kserve-controller-manager-776cc66d47-58vj8\" (UID: \"545fb734-bf40-43f8-a19d-0fe82840ef71\") " pod="opendatahub/kserve-controller-manager-776cc66d47-58vj8" Nov 04 03:55:11.143119 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.143079 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bffcd9a5-5579-4223-ad60-f1c5a862645f-tls-certs\") pod \"model-serving-api-75bd89fdfc-2m46k\" (UID: \"bffcd9a5-5579-4223-ad60-f1c5a862645f\") " pod="opendatahub/model-serving-api-75bd89fdfc-2m46k" Nov 04 03:55:11.143119 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.143107 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06eeac89-43b5-4e21-bd3c-e480330635e8-metrics-certs\") pod \"odh-model-controller-769875db56-xdp9r\" (UID: \"06eeac89-43b5-4e21-bd3c-e480330635e8\") " pod="opendatahub/odh-model-controller-769875db56-xdp9r" Nov 04 03:55:11.143398 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.143161 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzxch\" (UniqueName: \"kubernetes.io/projected/06eeac89-43b5-4e21-bd3c-e480330635e8-kube-api-access-mzxch\") pod \"odh-model-controller-769875db56-xdp9r\" (UID: \"06eeac89-43b5-4e21-bd3c-e480330635e8\") " pod="opendatahub/odh-model-controller-769875db56-xdp9r" Nov 04 03:55:11.143398 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.143226 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/545fb734-bf40-43f8-a19d-0fe82840ef71-cert\") pod \"kserve-controller-manager-776cc66d47-58vj8\" (UID: \"545fb734-bf40-43f8-a19d-0fe82840ef71\") " pod="opendatahub/kserve-controller-manager-776cc66d47-58vj8" Nov 04 03:55:11.143398 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.143246 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhpz4\" (UniqueName: \"kubernetes.io/projected/bffcd9a5-5579-4223-ad60-f1c5a862645f-kube-api-access-bhpz4\") pod \"model-serving-api-75bd89fdfc-2m46k\" (UID: \"bffcd9a5-5579-4223-ad60-f1c5a862645f\") " pod="opendatahub/model-serving-api-75bd89fdfc-2m46k" Nov 04 03:55:11.143398 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.143304 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06eeac89-43b5-4e21-bd3c-e480330635e8-cert\") pod \"odh-model-controller-769875db56-xdp9r\" (UID: \"06eeac89-43b5-4e21-bd3c-e480330635e8\") " pod="opendatahub/odh-model-controller-769875db56-xdp9r" Nov 04 03:55:11.145754 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.145731 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bffcd9a5-5579-4223-ad60-f1c5a862645f-tls-certs\") pod \"model-serving-api-75bd89fdfc-2m46k\" (UID: \"bffcd9a5-5579-4223-ad60-f1c5a862645f\") " pod="opendatahub/model-serving-api-75bd89fdfc-2m46k" Nov 04 03:55:11.145885 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.145845 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/545fb734-bf40-43f8-a19d-0fe82840ef71-cert\") pod \"kserve-controller-manager-776cc66d47-58vj8\" (UID: \"545fb734-bf40-43f8-a19d-0fe82840ef71\") " pod="opendatahub/kserve-controller-manager-776cc66d47-58vj8" Nov 04 03:55:11.155862 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.155832 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwc8x\" (UniqueName: \"kubernetes.io/projected/545fb734-bf40-43f8-a19d-0fe82840ef71-kube-api-access-xwc8x\") pod \"kserve-controller-manager-776cc66d47-58vj8\" (UID: \"545fb734-bf40-43f8-a19d-0fe82840ef71\") " pod="opendatahub/kserve-controller-manager-776cc66d47-58vj8" Nov 04 03:55:11.155862 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.155844 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhpz4\" (UniqueName: \"kubernetes.io/projected/bffcd9a5-5579-4223-ad60-f1c5a862645f-kube-api-access-bhpz4\") pod \"model-serving-api-75bd89fdfc-2m46k\" (UID: \"bffcd9a5-5579-4223-ad60-f1c5a862645f\") " pod="opendatahub/model-serving-api-75bd89fdfc-2m46k" Nov 04 03:55:11.198226 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.198195 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-58vj8" Nov 04 03:55:11.244469 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.244437 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06eeac89-43b5-4e21-bd3c-e480330635e8-cert\") pod \"odh-model-controller-769875db56-xdp9r\" (UID: \"06eeac89-43b5-4e21-bd3c-e480330635e8\") " pod="opendatahub/odh-model-controller-769875db56-xdp9r" Nov 04 03:55:11.244621 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.244506 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06eeac89-43b5-4e21-bd3c-e480330635e8-metrics-certs\") pod \"odh-model-controller-769875db56-xdp9r\" (UID: \"06eeac89-43b5-4e21-bd3c-e480330635e8\") " pod="opendatahub/odh-model-controller-769875db56-xdp9r" Nov 04 03:55:11.244621 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.244538 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzxch\" (UniqueName: \"kubernetes.io/projected/06eeac89-43b5-4e21-bd3c-e480330635e8-kube-api-access-mzxch\") pod \"odh-model-controller-769875db56-xdp9r\" (UID: \"06eeac89-43b5-4e21-bd3c-e480330635e8\") " pod="opendatahub/odh-model-controller-769875db56-xdp9r" Nov 04 03:55:11.246699 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.246668 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-2m46k" Nov 04 03:55:11.247746 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.247723 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06eeac89-43b5-4e21-bd3c-e480330635e8-metrics-certs\") pod \"odh-model-controller-769875db56-xdp9r\" (UID: \"06eeac89-43b5-4e21-bd3c-e480330635e8\") " pod="opendatahub/odh-model-controller-769875db56-xdp9r" Nov 04 03:55:11.247862 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.247739 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06eeac89-43b5-4e21-bd3c-e480330635e8-cert\") pod \"odh-model-controller-769875db56-xdp9r\" (UID: \"06eeac89-43b5-4e21-bd3c-e480330635e8\") " pod="opendatahub/odh-model-controller-769875db56-xdp9r" Nov 04 03:55:11.259717 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.259673 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzxch\" (UniqueName: \"kubernetes.io/projected/06eeac89-43b5-4e21-bd3c-e480330635e8-kube-api-access-mzxch\") pod \"odh-model-controller-769875db56-xdp9r\" (UID: \"06eeac89-43b5-4e21-bd3c-e480330635e8\") " pod="opendatahub/odh-model-controller-769875db56-xdp9r" Nov 04 03:55:11.272247 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.271663 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-xdp9r" Nov 04 03:55:11.482625 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.482598 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-58vj8"] Nov 04 03:55:11.484700 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:55:11.484670 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545fb734_bf40_43f8_a19d_0fe82840ef71.slice/crio-8af8bbf5b06ac6cfa3ba00816672059cccabbf379533b940e1328ea3404f2173 WatchSource:0}: Error finding container 8af8bbf5b06ac6cfa3ba00816672059cccabbf379533b940e1328ea3404f2173: Status 404 returned error can't find the container with id 8af8bbf5b06ac6cfa3ba00816672059cccabbf379533b940e1328ea3404f2173 Nov 04 03:55:11.553352 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.553322 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-2m46k"] Nov 04 03:55:11.555948 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:55:11.555887 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbffcd9a5_5579_4223_ad60_f1c5a862645f.slice/crio-fb9988e02b4b4a89ea47e425df79c0f47f92e390fad9ca70dc2a1502bebb8f8a WatchSource:0}: Error finding container fb9988e02b4b4a89ea47e425df79c0f47f92e390fad9ca70dc2a1502bebb8f8a: Status 404 returned error can't find the container with id fb9988e02b4b4a89ea47e425df79c0f47f92e390fad9ca70dc2a1502bebb8f8a Nov 04 03:55:11.582665 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:11.582643 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-xdp9r"] Nov 04 03:55:11.584461 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:55:11.584437 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06eeac89_43b5_4e21_bd3c_e480330635e8.slice/crio-61afd93957defdc0bb438114ce358ed669302016d5d6accc4e697ed3fa73c72e WatchSource:0}: Error finding container 61afd93957defdc0bb438114ce358ed669302016d5d6accc4e697ed3fa73c72e: Status 404 returned error can't find the container with id 61afd93957defdc0bb438114ce358ed669302016d5d6accc4e697ed3fa73c72e Nov 04 03:55:12.157683 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:12.157640 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-xdp9r" event={"ID":"06eeac89-43b5-4e21-bd3c-e480330635e8","Type":"ContainerStarted","Data":"61afd93957defdc0bb438114ce358ed669302016d5d6accc4e697ed3fa73c72e"} Nov 04 03:55:12.159699 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:12.159662 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-58vj8" event={"ID":"545fb734-bf40-43f8-a19d-0fe82840ef71","Type":"ContainerStarted","Data":"8af8bbf5b06ac6cfa3ba00816672059cccabbf379533b940e1328ea3404f2173"} Nov 04 03:55:12.162715 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:12.162679 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-2m46k" event={"ID":"bffcd9a5-5579-4223-ad60-f1c5a862645f","Type":"ContainerStarted","Data":"fb9988e02b4b4a89ea47e425df79c0f47f92e390fad9ca70dc2a1502bebb8f8a"} Nov 04 03:55:15.185158 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:15.185124 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-58vj8" event={"ID":"545fb734-bf40-43f8-a19d-0fe82840ef71","Type":"ContainerStarted","Data":"97168b5dd13e8dd6a8b21551afb92e41c6ab2e75021bccee2059584c9e620636"} Nov 04 03:55:15.185534 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:15.185195 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-58vj8" Nov 04 03:55:15.186824 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:15.186797 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-2m46k" event={"ID":"bffcd9a5-5579-4223-ad60-f1c5a862645f","Type":"ContainerStarted","Data":"efafa8053d1ab79bd57a109ad94173d5341ef6fe50d99e14cf7606858490ec58"} Nov 04 03:55:15.186941 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:15.186858 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-2m46k" Nov 04 03:55:15.188525 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:15.188504 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-xdp9r" event={"ID":"06eeac89-43b5-4e21-bd3c-e480330635e8","Type":"ContainerStarted","Data":"178a5db7093fc39d1153a6babcd2caf4c7a6f06fd94e19a5ecde32960d948fbc"} Nov 04 03:55:15.188608 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:15.188564 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-xdp9r" Nov 04 03:55:15.206679 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:15.206639 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-58vj8" podStartSLOduration=2.096367645 podStartE2EDuration="5.206628024s" totalImagesPullingTime="3.110260379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:55:15.205011413 +0000 UTC m=+359.005957221" watchObservedRunningTime="2026-11-04 03:55:15.206628024 +0000 UTC m=+359.007573860" Nov 04 03:55:15.227154 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:15.227110 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-xdp9r" podStartSLOduration=1.70707794 podStartE2EDuration="5.227095566s" totalImagesPullingTime="3.520017626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:55:15.226673258 +0000 UTC m=+359.027619070" watchObservedRunningTime="2026-11-04 03:55:15.227095566 +0000 UTC m=+359.028041368" Nov 04 03:55:15.246677 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:15.246638 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-2m46k" podStartSLOduration=2.50513767 podStartE2EDuration="5.246625221s" totalImagesPullingTime="2.741487551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:55:15.245478012 +0000 UTC m=+359.046423828" watchObservedRunningTime="2026-11-04 03:55:15.246625221 +0000 UTC m=+359.047571029" Nov 04 03:55:16.122493 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:16.122458 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55cc4d9668-f6wtl" Nov 04 03:55:16.175136 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:16.175106 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-xqc54"] Nov 04 03:55:16.175391 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:16.175361 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-8b95c9c66-xqc54" podUID="47f40d1b-c510-4696-bc31-0c792fd06de4" containerName="manager" containerID="cri-o://5cda23fc210b013990ff245fa57963fa520090e16e3d984028ce890868af5245" gracePeriod=30 Nov 04 03:55:16.177301 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:16.177274 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8b95c9c66-xqc54" Nov 04 03:55:16.425719 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:16.425695 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8b95c9c66-xqc54" Nov 04 03:55:16.492250 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:16.492215 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47f40d1b-c510-4696-bc31-0c792fd06de4-webhook-cert\" (UniqueName: \"kubernetes.io/secret/47f40d1b-c510-4696-bc31-0c792fd06de4-webhook-cert\") pod \"47f40d1b-c510-4696-bc31-0c792fd06de4\" (UID: \"47f40d1b-c510-4696-bc31-0c792fd06de4\") " Nov 04 03:55:16.492410 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:16.492272 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47f40d1b-c510-4696-bc31-0c792fd06de4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47f40d1b-c510-4696-bc31-0c792fd06de4-metrics-certs\") pod \"47f40d1b-c510-4696-bc31-0c792fd06de4\" (UID: \"47f40d1b-c510-4696-bc31-0c792fd06de4\") " Nov 04 03:55:16.492410 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:16.492327 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f40d1b-c510-4696-bc31-0c792fd06de4-kube-api-access-ctn4r\" (UniqueName: \"kubernetes.io/projected/47f40d1b-c510-4696-bc31-0c792fd06de4-kube-api-access-ctn4r\") pod \"47f40d1b-c510-4696-bc31-0c792fd06de4\" (UID: \"47f40d1b-c510-4696-bc31-0c792fd06de4\") " Nov 04 03:55:16.494489 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:16.494451 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47f40d1b-c510-4696-bc31-0c792fd06de4-metrics-certs" pod "47f40d1b-c510-4696-bc31-0c792fd06de4" (UID: "47f40d1b-c510-4696-bc31-0c792fd06de4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:55:16.494617 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:16.494497 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47f40d1b-c510-4696-bc31-0c792fd06de4-webhook-cert" pod "47f40d1b-c510-4696-bc31-0c792fd06de4" (UID: "47f40d1b-c510-4696-bc31-0c792fd06de4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:55:16.494617 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:16.494573 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f40d1b-c510-4696-bc31-0c792fd06de4-kube-api-access-ctn4r" pod "47f40d1b-c510-4696-bc31-0c792fd06de4" (UID: "47f40d1b-c510-4696-bc31-0c792fd06de4"). InnerVolumeSpecName "kube-api-access-ctn4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:55:16.593195 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:16.593164 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctn4r\" (UniqueName: \"kubernetes.io/projected/47f40d1b-c510-4696-bc31-0c792fd06de4-kube-api-access-ctn4r\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:55:16.593195 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:16.593192 4966 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47f40d1b-c510-4696-bc31-0c792fd06de4-webhook-cert\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:55:16.593195 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:16.593203 4966 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47f40d1b-c510-4696-bc31-0c792fd06de4-metrics-certs\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:55:17.199659 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:17.199622 4966 generic.go:356] "Generic (PLEG): container finished" podID="47f40d1b-c510-4696-bc31-0c792fd06de4" containerID="5cda23fc210b013990ff245fa57963fa520090e16e3d984028ce890868af5245" exitCode=0 Nov 04 03:55:17.199844 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:17.199678 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8b95c9c66-xqc54" Nov 04 03:55:17.199844 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:17.199702 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8b95c9c66-xqc54" event={"ID":"47f40d1b-c510-4696-bc31-0c792fd06de4","Type":"ContainerDied","Data":"5cda23fc210b013990ff245fa57963fa520090e16e3d984028ce890868af5245"} Nov 04 03:55:17.199844 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:17.199813 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8b95c9c66-xqc54" event={"ID":"47f40d1b-c510-4696-bc31-0c792fd06de4","Type":"ContainerDied","Data":"523d8e45273420504b6dd707756424333bc39d05562b2149b8160090753af6c7"} Nov 04 03:55:17.199844 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:17.199843 4966 scope.go:122] "RemoveContainer" containerID="5cda23fc210b013990ff245fa57963fa520090e16e3d984028ce890868af5245" Nov 04 03:55:17.209422 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:17.209402 4966 scope.go:122] "RemoveContainer" containerID="5cda23fc210b013990ff245fa57963fa520090e16e3d984028ce890868af5245" Nov 04 03:55:17.209683 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:55:17.209664 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cda23fc210b013990ff245fa57963fa520090e16e3d984028ce890868af5245\": container with ID starting with 5cda23fc210b013990ff245fa57963fa520090e16e3d984028ce890868af5245 not found: ID does not exist" containerID="5cda23fc210b013990ff245fa57963fa520090e16e3d984028ce890868af5245" Nov 04 03:55:17.209727 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:17.209691 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cda23fc210b013990ff245fa57963fa520090e16e3d984028ce890868af5245"} err="failed to get container status \"5cda23fc210b013990ff245fa57963fa520090e16e3d984028ce890868af5245\": rpc error: code = NotFound desc = could not find container \"5cda23fc210b013990ff245fa57963fa520090e16e3d984028ce890868af5245\": container with ID starting with 5cda23fc210b013990ff245fa57963fa520090e16e3d984028ce890868af5245 not found: ID does not exist" Nov 04 03:55:17.223886 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:17.223861 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-xqc54"] Nov 04 03:55:17.227270 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:17.227245 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-xqc54"] Nov 04 03:55:18.340273 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:18.340239 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47f40d1b-c510-4696-bc31-0c792fd06de4" path="/var/lib/kubelet/pods/47f40d1b-c510-4696-bc31-0c792fd06de4/volumes" Nov 04 03:55:26.197079 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:26.197047 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-xdp9r" Nov 04 03:55:26.198825 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:26.198804 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-2m46k" Nov 04 03:55:35.676221 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:55:35.676181 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:55:46.199645 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:46.199615 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-58vj8" Nov 04 03:55:48.658854 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.658820 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dd76875b5-6zvxx"] Nov 04 03:55:48.659259 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.659206 4966 cpu_manager.go:414] "containerMap: removing container" podUID="47f40d1b-c510-4696-bc31-0c792fd06de4" containerName="manager" Nov 04 03:55:48.659259 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.659217 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47f40d1b-c510-4696-bc31-0c792fd06de4" containerName="manager" Nov 04 03:55:48.659331 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.659295 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="47f40d1b-c510-4696-bc31-0c792fd06de4" containerName="manager" Nov 04 03:55:48.662484 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.662467 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-6zvxx" Nov 04 03:55:48.671601 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.671549 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jxzfk\"" Nov 04 03:55:48.671601 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.671593 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 04 03:55:48.672530 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.672504 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-6zvxx"] Nov 04 03:55:48.694257 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.694230 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-6zvxx"] Nov 04 03:55:48.694444 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:55:48.694424 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2xb7x tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-2xb7x tls-cert]: context canceled" pod="kuadrant-system/authorino-6dd76875b5-6zvxx" podUID="6fbed1da-4f6e-4c8e-a23c-2d7373bf7758" Nov 04 03:55:48.768327 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.768290 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6fbed1da-4f6e-4c8e-a23c-2d7373bf7758-tls-cert\") pod \"authorino-6dd76875b5-6zvxx\" (UID: \"6fbed1da-4f6e-4c8e-a23c-2d7373bf7758\") " pod="kuadrant-system/authorino-6dd76875b5-6zvxx" Nov 04 03:55:48.768493 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.768362 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xb7x\" (UniqueName: \"kubernetes.io/projected/6fbed1da-4f6e-4c8e-a23c-2d7373bf7758-kube-api-access-2xb7x\") pod \"authorino-6dd76875b5-6zvxx\" (UID: \"6fbed1da-4f6e-4c8e-a23c-2d7373bf7758\") " pod="kuadrant-system/authorino-6dd76875b5-6zvxx" Nov 04 03:55:48.860232 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.860193 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5df874889d-sr95h"] Nov 04 03:55:48.863817 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.863796 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5df874889d-sr95h" Nov 04 03:55:48.868615 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.868591 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:55:48.868848 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.868688 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:55:48.868963 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.868941 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6fbed1da-4f6e-4c8e-a23c-2d7373bf7758-tls-cert\") pod \"authorino-6dd76875b5-6zvxx\" (UID: \"6fbed1da-4f6e-4c8e-a23c-2d7373bf7758\") " pod="kuadrant-system/authorino-6dd76875b5-6zvxx" Nov 04 03:55:48.869022 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.869005 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xb7x\" (UniqueName: \"kubernetes.io/projected/6fbed1da-4f6e-4c8e-a23c-2d7373bf7758-kube-api-access-2xb7x\") pod \"authorino-6dd76875b5-6zvxx\" (UID: \"6fbed1da-4f6e-4c8e-a23c-2d7373bf7758\") " pod="kuadrant-system/authorino-6dd76875b5-6zvxx" Nov 04 03:55:48.870648 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.870622 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bwgrq\"" Nov 04 03:55:48.870772 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.870647 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 03:55:48.870772 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.870655 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 04 03:55:48.871837 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.871818 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6fbed1da-4f6e-4c8e-a23c-2d7373bf7758-tls-cert\") pod \"authorino-6dd76875b5-6zvxx\" (UID: \"6fbed1da-4f6e-4c8e-a23c-2d7373bf7758\") " pod="kuadrant-system/authorino-6dd76875b5-6zvxx" Nov 04 03:55:48.875189 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.875167 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5df874889d-sr95h"] Nov 04 03:55:48.881583 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.881560 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xb7x\" (UniqueName: \"kubernetes.io/projected/6fbed1da-4f6e-4c8e-a23c-2d7373bf7758-kube-api-access-2xb7x\") pod \"authorino-6dd76875b5-6zvxx\" (UID: \"6fbed1da-4f6e-4c8e-a23c-2d7373bf7758\") " pod="kuadrant-system/authorino-6dd76875b5-6zvxx" Nov 04 03:55:48.969973 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.969854 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e6b9d46-036a-4177-8583-c06b1bbba58b-openshift-service-ca\") pod \"maas-api-5df874889d-sr95h\" (UID: \"9e6b9d46-036a-4177-8583-c06b1bbba58b\") " pod="odh-ai-gateway-infra/maas-api-5df874889d-sr95h" Nov 04 03:55:48.969973 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.969912 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e6b9d46-036a-4177-8583-c06b1bbba58b-metrics-certs\") pod \"maas-api-5df874889d-sr95h\" (UID: \"9e6b9d46-036a-4177-8583-c06b1bbba58b\") " pod="odh-ai-gateway-infra/maas-api-5df874889d-sr95h" Nov 04 03:55:48.970169 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.970101 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e6b9d46-036a-4177-8583-c06b1bbba58b-maas-api-tls\") pod \"maas-api-5df874889d-sr95h\" (UID: \"9e6b9d46-036a-4177-8583-c06b1bbba58b\") " pod="odh-ai-gateway-infra/maas-api-5df874889d-sr95h" Nov 04 03:55:48.970212 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:48.970165 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tswpl\" (UniqueName: \"kubernetes.io/projected/9e6b9d46-036a-4177-8583-c06b1bbba58b-kube-api-access-tswpl\") pod \"maas-api-5df874889d-sr95h\" (UID: \"9e6b9d46-036a-4177-8583-c06b1bbba58b\") " pod="odh-ai-gateway-infra/maas-api-5df874889d-sr95h" Nov 04 03:55:49.070824 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:49.070785 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e6b9d46-036a-4177-8583-c06b1bbba58b-maas-api-tls\") pod \"maas-api-5df874889d-sr95h\" (UID: \"9e6b9d46-036a-4177-8583-c06b1bbba58b\") " pod="odh-ai-gateway-infra/maas-api-5df874889d-sr95h" Nov 04 03:55:49.071022 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:49.070831 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tswpl\" (UniqueName: \"kubernetes.io/projected/9e6b9d46-036a-4177-8583-c06b1bbba58b-kube-api-access-tswpl\") pod \"maas-api-5df874889d-sr95h\" (UID: \"9e6b9d46-036a-4177-8583-c06b1bbba58b\") " pod="odh-ai-gateway-infra/maas-api-5df874889d-sr95h" Nov 04 03:55:49.071022 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:49.070880 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e6b9d46-036a-4177-8583-c06b1bbba58b-openshift-service-ca\") pod \"maas-api-5df874889d-sr95h\" (UID: \"9e6b9d46-036a-4177-8583-c06b1bbba58b\") " pod="odh-ai-gateway-infra/maas-api-5df874889d-sr95h" Nov 04 03:55:49.071022 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:49.070897 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e6b9d46-036a-4177-8583-c06b1bbba58b-metrics-certs\") pod \"maas-api-5df874889d-sr95h\" (UID: \"9e6b9d46-036a-4177-8583-c06b1bbba58b\") " pod="odh-ai-gateway-infra/maas-api-5df874889d-sr95h" Nov 04 03:55:49.071647 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:49.071623 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e6b9d46-036a-4177-8583-c06b1bbba58b-openshift-service-ca\") pod \"maas-api-5df874889d-sr95h\" (UID: \"9e6b9d46-036a-4177-8583-c06b1bbba58b\") " pod="odh-ai-gateway-infra/maas-api-5df874889d-sr95h" Nov 04 03:55:49.073310 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:49.073288 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e6b9d46-036a-4177-8583-c06b1bbba58b-metrics-certs\") pod \"maas-api-5df874889d-sr95h\" (UID: \"9e6b9d46-036a-4177-8583-c06b1bbba58b\") " pod="odh-ai-gateway-infra/maas-api-5df874889d-sr95h" Nov 04 03:55:49.073477 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:49.073455 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e6b9d46-036a-4177-8583-c06b1bbba58b-maas-api-tls\") pod \"maas-api-5df874889d-sr95h\" (UID: \"9e6b9d46-036a-4177-8583-c06b1bbba58b\") " pod="odh-ai-gateway-infra/maas-api-5df874889d-sr95h" Nov 04 03:55:49.088239 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:49.088209 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tswpl\" (UniqueName: \"kubernetes.io/projected/9e6b9d46-036a-4177-8583-c06b1bbba58b-kube-api-access-tswpl\") pod \"maas-api-5df874889d-sr95h\" (UID: \"9e6b9d46-036a-4177-8583-c06b1bbba58b\") " pod="odh-ai-gateway-infra/maas-api-5df874889d-sr95h" Nov 04 03:55:49.188141 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:49.188100 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5df874889d-sr95h" Nov 04 03:55:49.362942 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:49.362889 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-6zvxx" Nov 04 03:55:49.371289 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:49.371266 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-6zvxx" Nov 04 03:55:49.466227 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:49.466204 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5df874889d-sr95h"] Nov 04 03:55:49.468143 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:55:49.468109 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6b9d46_036a_4177_8583_c06b1bbba58b.slice/crio-259fc87c54becc5457a4adcaec93c2e262b2304d39056866348278c7b1c53266 WatchSource:0}: Error finding container 259fc87c54becc5457a4adcaec93c2e262b2304d39056866348278c7b1c53266: Status 404 returned error can't find the container with id 259fc87c54becc5457a4adcaec93c2e262b2304d39056866348278c7b1c53266 Nov 04 03:55:49.475717 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:49.475693 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fbed1da-4f6e-4c8e-a23c-2d7373bf7758-tls-cert\" (UniqueName: \"kubernetes.io/secret/6fbed1da-4f6e-4c8e-a23c-2d7373bf7758-tls-cert\") pod \"6fbed1da-4f6e-4c8e-a23c-2d7373bf7758\" (UID: \"6fbed1da-4f6e-4c8e-a23c-2d7373bf7758\") " Nov 04 03:55:49.475820 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:49.475753 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fbed1da-4f6e-4c8e-a23c-2d7373bf7758-kube-api-access-2xb7x\" (UniqueName: \"kubernetes.io/projected/6fbed1da-4f6e-4c8e-a23c-2d7373bf7758-kube-api-access-2xb7x\") pod \"6fbed1da-4f6e-4c8e-a23c-2d7373bf7758\" (UID: \"6fbed1da-4f6e-4c8e-a23c-2d7373bf7758\") " Nov 04 03:55:49.477829 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:49.477804 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fbed1da-4f6e-4c8e-a23c-2d7373bf7758-kube-api-access-2xb7x" pod "6fbed1da-4f6e-4c8e-a23c-2d7373bf7758" (UID: "6fbed1da-4f6e-4c8e-a23c-2d7373bf7758"). InnerVolumeSpecName "kube-api-access-2xb7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:55:49.477829 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:49.477808 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fbed1da-4f6e-4c8e-a23c-2d7373bf7758-tls-cert" pod "6fbed1da-4f6e-4c8e-a23c-2d7373bf7758" (UID: "6fbed1da-4f6e-4c8e-a23c-2d7373bf7758"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:55:49.577041 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:49.577013 4966 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6fbed1da-4f6e-4c8e-a23c-2d7373bf7758-tls-cert\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:55:49.577041 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:49.577037 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xb7x\" (UniqueName: \"kubernetes.io/projected/6fbed1da-4f6e-4c8e-a23c-2d7373bf7758-kube-api-access-2xb7x\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 03:55:50.368776 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:50.368738 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5df874889d-sr95h" event={"ID":"9e6b9d46-036a-4177-8583-c06b1bbba58b","Type":"ContainerStarted","Data":"259fc87c54becc5457a4adcaec93c2e262b2304d39056866348278c7b1c53266"} Nov 04 03:55:50.369200 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:50.368777 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-6zvxx" Nov 04 03:55:50.403068 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:50.403036 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-6zvxx"] Nov 04 03:55:50.410875 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:50.410842 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-6zvxx"] Nov 04 03:55:51.375275 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:51.375244 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5df874889d-sr95h" event={"ID":"9e6b9d46-036a-4177-8583-c06b1bbba58b","Type":"ContainerStarted","Data":"cb2bc6488621a7c5a5d9786dc145e4b9185e3ca7b10dc747f0badb44b7b371b4"} Nov 04 03:55:51.375678 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:51.375310 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5df874889d-sr95h" Nov 04 03:55:51.401143 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:51.401086 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5df874889d-sr95h" podStartSLOduration=1.589995778 podStartE2EDuration="3.401073172s" totalImagesPullingTime="1.811077394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:55:51.400412035 +0000 UTC m=+395.201357847" watchObservedRunningTime="2026-11-04 03:55:51.401073172 +0000 UTC m=+395.202018979" Nov 04 03:55:52.340489 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:52.340456 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fbed1da-4f6e-4c8e-a23c-2d7373bf7758" path="/var/lib/kubelet/pods/6fbed1da-4f6e-4c8e-a23c-2d7373bf7758/volumes" Nov 04 03:55:57.385725 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:55:57.385693 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5df874889d-sr95h" Nov 04 03:56:05.676755 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:56:05.676675 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:56:35.676941 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:56:35.676890 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:57:05.677108 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:57:05.677062 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:57:07.913710 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:07.913677 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv"] Nov 04 03:57:07.917142 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:07.917124 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:07.926613 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:07.926589 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 04 03:57:07.926718 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:07.926651 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-68php\"" Nov 04 03:57:07.928246 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:07.928231 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 04 03:57:07.928306 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:07.928280 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 04 03:57:07.931975 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:07.931951 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv"] Nov 04 03:57:07.969995 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:07.969962 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7b4b6c-19ba-445e-bd66-5292b0a28b0e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv\" (UID: \"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:07.970155 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:07.970100 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7b4b6c-19ba-445e-bd66-5292b0a28b0e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv\" (UID: \"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:07.970155 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:07.970140 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7b4b6c-19ba-445e-bd66-5292b0a28b0e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv\" (UID: \"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:07.970282 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:07.970244 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7b4b6c-19ba-445e-bd66-5292b0a28b0e-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv\" (UID: \"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:07.970365 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:07.970347 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7b4b6c-19ba-445e-bd66-5292b0a28b0e-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv\" (UID: \"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:07.970422 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:07.970385 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb5cr\" (UniqueName: \"kubernetes.io/projected/6f7b4b6c-19ba-445e-bd66-5292b0a28b0e-kube-api-access-pb5cr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv\" (UID: \"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:08.071026 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:08.070985 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7b4b6c-19ba-445e-bd66-5292b0a28b0e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv\" (UID: \"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:08.071026 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:08.071030 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7b4b6c-19ba-445e-bd66-5292b0a28b0e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv\" (UID: \"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:08.071273 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:08.071077 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7b4b6c-19ba-445e-bd66-5292b0a28b0e-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv\" (UID: \"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:08.071273 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:08.071131 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7b4b6c-19ba-445e-bd66-5292b0a28b0e-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv\" (UID: \"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:08.071273 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:08.071159 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb5cr\" (UniqueName: \"kubernetes.io/projected/6f7b4b6c-19ba-445e-bd66-5292b0a28b0e-kube-api-access-pb5cr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv\" (UID: \"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:08.071273 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:08.071203 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7b4b6c-19ba-445e-bd66-5292b0a28b0e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv\" (UID: \"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:08.071465 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:08.071408 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7b4b6c-19ba-445e-bd66-5292b0a28b0e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv\" (UID: \"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:08.071540 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:08.071477 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7b4b6c-19ba-445e-bd66-5292b0a28b0e-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv\" (UID: \"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:08.071607 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:08.071594 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7b4b6c-19ba-445e-bd66-5292b0a28b0e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv\" (UID: \"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:08.073504 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:08.073477 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7b4b6c-19ba-445e-bd66-5292b0a28b0e-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv\" (UID: \"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:08.073620 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:08.073555 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7b4b6c-19ba-445e-bd66-5292b0a28b0e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv\" (UID: \"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:08.086873 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:08.086845 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb5cr\" (UniqueName: \"kubernetes.io/projected/6f7b4b6c-19ba-445e-bd66-5292b0a28b0e-kube-api-access-pb5cr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv\" (UID: \"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:08.230232 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:08.230145 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:08.507778 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:08.507686 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv"] Nov 04 03:57:08.510948 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:57:08.510905 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7b4b6c_19ba_445e_bd66_5292b0a28b0e.slice/crio-2e6e64bf9e79124474d6425e540d202d03d0985023806ee69f3335b08bc56170 WatchSource:0}: Error finding container 2e6e64bf9e79124474d6425e540d202d03d0985023806ee69f3335b08bc56170: Status 404 returned error can't find the container with id 2e6e64bf9e79124474d6425e540d202d03d0985023806ee69f3335b08bc56170 Nov 04 03:57:08.762909 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:08.762832 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" event={"ID":"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e","Type":"ContainerStarted","Data":"2e6e64bf9e79124474d6425e540d202d03d0985023806ee69f3335b08bc56170"} Nov 04 03:57:09.770165 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:09.770122 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" event={"ID":"6f7b4b6c-19ba-445e-bd66-5292b0a28b0e","Type":"ContainerStarted","Data":"88900fe112319460a77c998f30c0299f6cf34d617632d4940864afdeaafe3303"} Nov 04 03:57:09.792398 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:09.792322 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" podStartSLOduration=1.674954122 podStartE2EDuration="2.792307041s" totalImagesPullingTime="1.117352919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:57:09.790961711 +0000 UTC m=+473.591907516" watchObservedRunningTime="2026-11-04 03:57:09.792307041 +0000 UTC m=+473.593252849" Nov 04 03:57:09.908648 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:09.908567 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh"] Nov 04 03:57:09.912184 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:09.912162 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:09.920173 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:09.920150 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 04 03:57:09.922546 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:09.922521 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh"] Nov 04 03:57:09.990679 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:09.990636 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbx56\" (UniqueName: \"kubernetes.io/projected/ba477b57-6755-44f3-b42f-595bcf525d4a-kube-api-access-bbx56\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjthh\" (UID: \"ba477b57-6755-44f3-b42f-595bcf525d4a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:09.990842 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:09.990700 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba477b57-6755-44f3-b42f-595bcf525d4a-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjthh\" (UID: \"ba477b57-6755-44f3-b42f-595bcf525d4a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:09.990842 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:09.990796 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba477b57-6755-44f3-b42f-595bcf525d4a-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjthh\" (UID: \"ba477b57-6755-44f3-b42f-595bcf525d4a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:09.990842 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:09.990825 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba477b57-6755-44f3-b42f-595bcf525d4a-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjthh\" (UID: \"ba477b57-6755-44f3-b42f-595bcf525d4a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:09.990993 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:09.990862 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba477b57-6755-44f3-b42f-595bcf525d4a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjthh\" (UID: \"ba477b57-6755-44f3-b42f-595bcf525d4a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:09.990993 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:09.990912 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba477b57-6755-44f3-b42f-595bcf525d4a-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjthh\" (UID: \"ba477b57-6755-44f3-b42f-595bcf525d4a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:10.092319 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:10.092280 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba477b57-6755-44f3-b42f-595bcf525d4a-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjthh\" (UID: \"ba477b57-6755-44f3-b42f-595bcf525d4a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:10.092319 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:10.092317 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba477b57-6755-44f3-b42f-595bcf525d4a-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjthh\" (UID: \"ba477b57-6755-44f3-b42f-595bcf525d4a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:10.092559 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:10.092332 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba477b57-6755-44f3-b42f-595bcf525d4a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjthh\" (UID: \"ba477b57-6755-44f3-b42f-595bcf525d4a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:10.092559 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:10.092348 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba477b57-6755-44f3-b42f-595bcf525d4a-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjthh\" (UID: \"ba477b57-6755-44f3-b42f-595bcf525d4a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:10.092559 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:10.092418 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbx56\" (UniqueName: \"kubernetes.io/projected/ba477b57-6755-44f3-b42f-595bcf525d4a-kube-api-access-bbx56\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjthh\" (UID: \"ba477b57-6755-44f3-b42f-595bcf525d4a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:10.092559 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:10.092455 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba477b57-6755-44f3-b42f-595bcf525d4a-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjthh\" (UID: \"ba477b57-6755-44f3-b42f-595bcf525d4a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:10.092816 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:10.092788 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba477b57-6755-44f3-b42f-595bcf525d4a-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjthh\" (UID: \"ba477b57-6755-44f3-b42f-595bcf525d4a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:10.092955 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:10.092835 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba477b57-6755-44f3-b42f-595bcf525d4a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjthh\" (UID: \"ba477b57-6755-44f3-b42f-595bcf525d4a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:10.092955 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:10.092875 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba477b57-6755-44f3-b42f-595bcf525d4a-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjthh\" (UID: \"ba477b57-6755-44f3-b42f-595bcf525d4a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:10.095363 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:10.095335 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba477b57-6755-44f3-b42f-595bcf525d4a-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjthh\" (UID: \"ba477b57-6755-44f3-b42f-595bcf525d4a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:10.095505 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:10.095489 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba477b57-6755-44f3-b42f-595bcf525d4a-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjthh\" (UID: \"ba477b57-6755-44f3-b42f-595bcf525d4a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:10.105661 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:10.105638 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbx56\" (UniqueName: \"kubernetes.io/projected/ba477b57-6755-44f3-b42f-595bcf525d4a-kube-api-access-bbx56\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjthh\" (UID: \"ba477b57-6755-44f3-b42f-595bcf525d4a\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:10.227259 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:10.227171 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:10.231252 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:10.231234 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:10.503743 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:10.503714 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh"] Nov 04 03:57:10.505374 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:57:10.505347 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba477b57_6755_44f3_b42f_595bcf525d4a.slice/crio-a60b2972509ccfff44c15ee84dd0a8a77e69440f5d340669624900790e66e65f WatchSource:0}: Error finding container a60b2972509ccfff44c15ee84dd0a8a77e69440f5d340669624900790e66e65f: Status 404 returned error can't find the container with id a60b2972509ccfff44c15ee84dd0a8a77e69440f5d340669624900790e66e65f Nov 04 03:57:10.776823 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:10.776745 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" event={"ID":"ba477b57-6755-44f3-b42f-595bcf525d4a","Type":"ContainerStarted","Data":"a60b2972509ccfff44c15ee84dd0a8a77e69440f5d340669624900790e66e65f"} Nov 04 03:57:11.784111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:11.784074 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" event={"ID":"ba477b57-6755-44f3-b42f-595bcf525d4a","Type":"ContainerStarted","Data":"7e7814805771f93fdb7cbacb80eecb7411729f9c4981195618f11cf752126278"} Nov 04 03:57:11.815868 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:11.815809 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" podStartSLOduration=2.532694609 podStartE2EDuration="2.815794131s" totalImagesPullingTime="283.099522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:57:11.814011965 +0000 UTC m=+475.614957774" watchObservedRunningTime="2026-11-04 03:57:11.815794131 +0000 UTC m=+475.616739938" Nov 04 03:57:12.227742 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:12.227713 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:15.044852 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.044813 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b"] Nov 04 03:57:15.048859 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.048838 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.058618 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.058594 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 04 03:57:15.061997 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.061972 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b"] Nov 04 03:57:15.140256 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.140223 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3871362e-919f-4375-8030-cc85243036a4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6gs5b\" (UID: \"3871362e-919f-4375-8030-cc85243036a4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.140256 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.140258 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3871362e-919f-4375-8030-cc85243036a4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6gs5b\" (UID: \"3871362e-919f-4375-8030-cc85243036a4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.140463 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.140287 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3871362e-919f-4375-8030-cc85243036a4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6gs5b\" (UID: \"3871362e-919f-4375-8030-cc85243036a4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.140463 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.140418 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3871362e-919f-4375-8030-cc85243036a4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6gs5b\" (UID: \"3871362e-919f-4375-8030-cc85243036a4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.140463 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.140438 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3871362e-919f-4375-8030-cc85243036a4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6gs5b\" (UID: \"3871362e-919f-4375-8030-cc85243036a4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.140602 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.140474 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t22l\" (UniqueName: \"kubernetes.io/projected/3871362e-919f-4375-8030-cc85243036a4-kube-api-access-8t22l\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6gs5b\" (UID: \"3871362e-919f-4375-8030-cc85243036a4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.241753 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.241713 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3871362e-919f-4375-8030-cc85243036a4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6gs5b\" (UID: \"3871362e-919f-4375-8030-cc85243036a4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.241963 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.241774 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3871362e-919f-4375-8030-cc85243036a4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6gs5b\" (UID: \"3871362e-919f-4375-8030-cc85243036a4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.241963 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.241797 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3871362e-919f-4375-8030-cc85243036a4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6gs5b\" (UID: \"3871362e-919f-4375-8030-cc85243036a4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.241963 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.241844 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t22l\" (UniqueName: \"kubernetes.io/projected/3871362e-919f-4375-8030-cc85243036a4-kube-api-access-8t22l\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6gs5b\" (UID: \"3871362e-919f-4375-8030-cc85243036a4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.242176 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.242151 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3871362e-919f-4375-8030-cc85243036a4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6gs5b\" (UID: \"3871362e-919f-4375-8030-cc85243036a4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.242237 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.242197 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3871362e-919f-4375-8030-cc85243036a4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6gs5b\" (UID: \"3871362e-919f-4375-8030-cc85243036a4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.242237 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.242202 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3871362e-919f-4375-8030-cc85243036a4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6gs5b\" (UID: \"3871362e-919f-4375-8030-cc85243036a4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.242237 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.242219 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3871362e-919f-4375-8030-cc85243036a4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6gs5b\" (UID: \"3871362e-919f-4375-8030-cc85243036a4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.242464 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.242444 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3871362e-919f-4375-8030-cc85243036a4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6gs5b\" (UID: \"3871362e-919f-4375-8030-cc85243036a4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.244198 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.244174 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3871362e-919f-4375-8030-cc85243036a4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6gs5b\" (UID: \"3871362e-919f-4375-8030-cc85243036a4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.244484 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.244464 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3871362e-919f-4375-8030-cc85243036a4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6gs5b\" (UID: \"3871362e-919f-4375-8030-cc85243036a4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.255510 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.255480 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t22l\" (UniqueName: \"kubernetes.io/projected/3871362e-919f-4375-8030-cc85243036a4-kube-api-access-8t22l\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6gs5b\" (UID: \"3871362e-919f-4375-8030-cc85243036a4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.363974 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.363939 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:15.641673 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.641644 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b"] Nov 04 03:57:15.643478 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:57:15.643452 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3871362e_919f_4375_8030_cc85243036a4.slice/crio-a0c878cf72cf9d3d852662cb692586ae36300ed89e33db4414cc24f63a2b8fcb WatchSource:0}: Error finding container a0c878cf72cf9d3d852662cb692586ae36300ed89e33db4414cc24f63a2b8fcb: Status 404 returned error can't find the container with id a0c878cf72cf9d3d852662cb692586ae36300ed89e33db4414cc24f63a2b8fcb Nov 04 03:57:15.806466 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:15.806428 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" event={"ID":"3871362e-919f-4375-8030-cc85243036a4","Type":"ContainerStarted","Data":"a0c878cf72cf9d3d852662cb692586ae36300ed89e33db4414cc24f63a2b8fcb"} Nov 04 03:57:16.813528 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:16.813481 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" event={"ID":"3871362e-919f-4375-8030-cc85243036a4","Type":"ContainerStarted","Data":"65b7d8acf3572c6ad7133a54e5086b404b1d6d551e811ec684d3c3fcd789ca8c"} Nov 04 03:57:16.840174 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:16.840123 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" podStartSLOduration=1.558605373 podStartE2EDuration="1.840108831s" totalImagesPullingTime="281.503458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:57:16.838699925 +0000 UTC m=+480.639645734" watchObservedRunningTime="2026-11-04 03:57:16.840108831 +0000 UTC m=+480.641054676" Nov 04 03:57:17.364161 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:17.364134 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:18.230842 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:18.230805 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:18.243689 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:18.243664 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:18.829821 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:18.829788 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c"] Nov 04 03:57:18.836838 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:18.836810 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv" Nov 04 03:57:18.837021 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:18.836988 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:18.845993 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:18.845957 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 04 03:57:18.848275 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:18.848241 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c"] Nov 04 03:57:18.977244 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:18.977209 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6353ea56-85d9-422d-90e1-580b08d86d5f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c\" (UID: \"6353ea56-85d9-422d-90e1-580b08d86d5f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:18.977426 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:18.977256 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6353ea56-85d9-422d-90e1-580b08d86d5f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c\" (UID: \"6353ea56-85d9-422d-90e1-580b08d86d5f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:18.977426 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:18.977413 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6353ea56-85d9-422d-90e1-580b08d86d5f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c\" (UID: \"6353ea56-85d9-422d-90e1-580b08d86d5f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:18.977601 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:18.977514 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6353ea56-85d9-422d-90e1-580b08d86d5f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c\" (UID: \"6353ea56-85d9-422d-90e1-580b08d86d5f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:18.977601 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:18.977536 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6353ea56-85d9-422d-90e1-580b08d86d5f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c\" (UID: \"6353ea56-85d9-422d-90e1-580b08d86d5f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:18.977601 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:18.977580 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j7tb\" (UniqueName: \"kubernetes.io/projected/6353ea56-85d9-422d-90e1-580b08d86d5f-kube-api-access-4j7tb\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c\" (UID: \"6353ea56-85d9-422d-90e1-580b08d86d5f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:19.078561 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:19.078512 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j7tb\" (UniqueName: \"kubernetes.io/projected/6353ea56-85d9-422d-90e1-580b08d86d5f-kube-api-access-4j7tb\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c\" (UID: \"6353ea56-85d9-422d-90e1-580b08d86d5f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:19.078721 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:19.078590 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6353ea56-85d9-422d-90e1-580b08d86d5f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c\" (UID: \"6353ea56-85d9-422d-90e1-580b08d86d5f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:19.078721 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:19.078627 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6353ea56-85d9-422d-90e1-580b08d86d5f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c\" (UID: \"6353ea56-85d9-422d-90e1-580b08d86d5f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:19.078721 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:19.078681 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6353ea56-85d9-422d-90e1-580b08d86d5f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c\" (UID: \"6353ea56-85d9-422d-90e1-580b08d86d5f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:19.078857 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:19.078742 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6353ea56-85d9-422d-90e1-580b08d86d5f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c\" (UID: \"6353ea56-85d9-422d-90e1-580b08d86d5f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:19.078857 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:19.078769 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6353ea56-85d9-422d-90e1-580b08d86d5f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c\" (UID: \"6353ea56-85d9-422d-90e1-580b08d86d5f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:19.079131 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:19.079105 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6353ea56-85d9-422d-90e1-580b08d86d5f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c\" (UID: \"6353ea56-85d9-422d-90e1-580b08d86d5f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:19.079131 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:19.079120 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6353ea56-85d9-422d-90e1-580b08d86d5f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c\" (UID: \"6353ea56-85d9-422d-90e1-580b08d86d5f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:19.079258 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:19.079162 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6353ea56-85d9-422d-90e1-580b08d86d5f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c\" (UID: \"6353ea56-85d9-422d-90e1-580b08d86d5f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:19.081073 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:19.081027 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6353ea56-85d9-422d-90e1-580b08d86d5f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c\" (UID: \"6353ea56-85d9-422d-90e1-580b08d86d5f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:19.081322 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:19.081305 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6353ea56-85d9-422d-90e1-580b08d86d5f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c\" (UID: \"6353ea56-85d9-422d-90e1-580b08d86d5f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:19.093011 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:19.092985 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j7tb\" (UniqueName: \"kubernetes.io/projected/6353ea56-85d9-422d-90e1-580b08d86d5f-kube-api-access-4j7tb\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c\" (UID: \"6353ea56-85d9-422d-90e1-580b08d86d5f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:19.151929 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:19.151895 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:19.430110 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:19.430084 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c"] Nov 04 03:57:19.431954 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:57:19.431912 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6353ea56_85d9_422d_90e1_580b08d86d5f.slice/crio-6d77c4e05ed99b2124393c3a188b7391e6d786c11525e2bc34a26cf0b00f0450 WatchSource:0}: Error finding container 6d77c4e05ed99b2124393c3a188b7391e6d786c11525e2bc34a26cf0b00f0450: Status 404 returned error can't find the container with id 6d77c4e05ed99b2124393c3a188b7391e6d786c11525e2bc34a26cf0b00f0450 Nov 04 03:57:19.829988 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:19.829951 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" event={"ID":"6353ea56-85d9-422d-90e1-580b08d86d5f","Type":"ContainerStarted","Data":"2d09bf62e2a1488bea4d5228a54d91eb3d907a7a93f147657f91e6715b10bce6"} Nov 04 03:57:19.830186 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:19.830010 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" event={"ID":"6353ea56-85d9-422d-90e1-580b08d86d5f","Type":"ContainerStarted","Data":"6d77c4e05ed99b2124393c3a188b7391e6d786c11525e2bc34a26cf0b00f0450"} Nov 04 03:57:19.876294 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:19.876216 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" podStartSLOduration=1.575058794 podStartE2EDuration="1.876202182s" totalImagesPullingTime="301.143388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:57:19.875543592 +0000 UTC m=+483.676489399" watchObservedRunningTime="2026-11-04 03:57:19.876202182 +0000 UTC m=+483.677147989" Nov 04 03:57:20.152825 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:20.152792 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:20.227897 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:20.227864 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:20.240734 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:20.240712 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:20.847233 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:20.847206 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjthh" Nov 04 03:57:25.364778 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:25.364741 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:25.377516 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:25.377488 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:25.872940 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:25.872901 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6gs5b" Nov 04 03:57:27.045841 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.045804 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8"] Nov 04 03:57:27.049752 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.049731 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.054852 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.054831 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mb97r\"" Nov 04 03:57:27.062454 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.062431 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8"] Nov 04 03:57:27.163204 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.163169 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177f8aff-8b11-4891-9648-9c7aeb9305b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.163374 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.163240 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f8aff-8b11-4891-9648-9c7aeb9305b7-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.163374 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.163274 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177f8aff-8b11-4891-9648-9c7aeb9305b7-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.163492 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.163409 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177f8aff-8b11-4891-9648-9c7aeb9305b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.163538 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.163496 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f8aff-8b11-4891-9648-9c7aeb9305b7-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.163538 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.163525 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177f8aff-8b11-4891-9648-9c7aeb9305b7-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.163621 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.163552 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/177f8aff-8b11-4891-9648-9c7aeb9305b7-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.163662 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.163630 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177f8aff-8b11-4891-9648-9c7aeb9305b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.163711 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.163697 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc6jn\" (UniqueName: \"kubernetes.io/projected/177f8aff-8b11-4891-9648-9c7aeb9305b7-kube-api-access-qc6jn\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.265074 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.265045 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/177f8aff-8b11-4891-9648-9c7aeb9305b7-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.265074 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.265076 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177f8aff-8b11-4891-9648-9c7aeb9305b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.265267 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.265095 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc6jn\" (UniqueName: \"kubernetes.io/projected/177f8aff-8b11-4891-9648-9c7aeb9305b7-kube-api-access-qc6jn\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.265267 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.265128 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177f8aff-8b11-4891-9648-9c7aeb9305b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.265267 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.265166 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f8aff-8b11-4891-9648-9c7aeb9305b7-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.265267 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.265182 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177f8aff-8b11-4891-9648-9c7aeb9305b7-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.265267 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.265206 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177f8aff-8b11-4891-9648-9c7aeb9305b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.265267 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.265242 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f8aff-8b11-4891-9648-9c7aeb9305b7-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.265594 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.265274 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177f8aff-8b11-4891-9648-9c7aeb9305b7-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.265658 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.265615 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f8aff-8b11-4891-9648-9c7aeb9305b7-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.265710 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.265681 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177f8aff-8b11-4891-9648-9c7aeb9305b7-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.265763 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.265736 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177f8aff-8b11-4891-9648-9c7aeb9305b7-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.265843 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.265823 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f8aff-8b11-4891-9648-9c7aeb9305b7-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.266136 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.266114 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177f8aff-8b11-4891-9648-9c7aeb9305b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.267477 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.267449 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177f8aff-8b11-4891-9648-9c7aeb9305b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.267560 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.267540 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177f8aff-8b11-4891-9648-9c7aeb9305b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.277782 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.277758 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc6jn\" (UniqueName: \"kubernetes.io/projected/177f8aff-8b11-4891-9648-9c7aeb9305b7-kube-api-access-qc6jn\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.277951 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.277908 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/177f8aff-8b11-4891-9648-9c7aeb9305b7-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-4tpt8\" (UID: \"177f8aff-8b11-4891-9648-9c7aeb9305b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.365483 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.365455 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:27.647336 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.647302 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8"] Nov 04 03:57:27.649267 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:57:27.649241 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177f8aff_8b11_4891_9648_9c7aeb9305b7.slice/crio-cf46e9821fa648a90d40da5eb479e798637e199b5ef5c7566a39a6890a193d81 WatchSource:0}: Error finding container cf46e9821fa648a90d40da5eb479e798637e199b5ef5c7566a39a6890a193d81: Status 404 returned error can't find the container with id cf46e9821fa648a90d40da5eb479e798637e199b5ef5c7566a39a6890a193d81 Nov 04 03:57:27.651785 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.651752 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 03:57:27.651863 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.651823 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 03:57:27.651905 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.651859 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 03:57:27.875430 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.875395 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" event={"ID":"177f8aff-8b11-4891-9648-9c7aeb9305b7","Type":"ContainerStarted","Data":"3dceec7d2484d387daec20e4a7269793ced1ec4201fe4deb67785681f5960672"} Nov 04 03:57:27.875590 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.875452 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" event={"ID":"177f8aff-8b11-4891-9648-9c7aeb9305b7","Type":"ContainerStarted","Data":"cf46e9821fa648a90d40da5eb479e798637e199b5ef5c7566a39a6890a193d81"} Nov 04 03:57:27.902364 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:27.902269 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" podStartSLOduration=0.902252032 podStartE2EDuration="902.252032ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:57:27.899348528 +0000 UTC m=+491.700294348" watchObservedRunningTime="2026-11-04 03:57:27.902252032 +0000 UTC m=+491.703197840" Nov 04 03:57:28.374008 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:28.373972 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:28.379529 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:28.379496 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 04 03:57:28.379692 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:28.379557 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" podUID="177f8aff-8b11-4891-9648-9c7aeb9305b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 04 03:57:29.152904 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:29.152871 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:29.173571 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:29.173541 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:29.377045 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:29.377006 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 04 03:57:29.377605 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:29.377571 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" podUID="177f8aff-8b11-4891-9648-9c7aeb9305b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 04 03:57:29.900650 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:29.900618 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c" Nov 04 03:57:30.368396 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:30.368354 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 04 03:57:30.368700 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:30.368665 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" podUID="177f8aff-8b11-4891-9648-9c7aeb9305b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 04 03:57:31.373367 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:31.373328 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 04 03:57:31.373971 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:31.373909 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" podUID="177f8aff-8b11-4891-9648-9c7aeb9305b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 04 03:57:32.371328 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:32.371290 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 04 03:57:32.371608 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:32.371573 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" podUID="177f8aff-8b11-4891-9648-9c7aeb9305b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 04 03:57:32.672932 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:32.672816 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-cbbff8c7f-crdpg"] Nov 04 03:57:32.688209 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:32.688173 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cbbff8c7f-crdpg"] Nov 04 03:57:32.688392 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:32.688292 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cbbff8c7f-crdpg" Nov 04 03:57:32.695556 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:32.695529 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 04 03:57:32.695681 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:32.695605 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jxzfk\"" Nov 04 03:57:32.825690 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:32.825645 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnlk8\" (UniqueName: \"kubernetes.io/projected/dec9abf1-8b69-46d3-b5da-cb0b552b8447-kube-api-access-qnlk8\") pod \"authorino-cbbff8c7f-crdpg\" (UID: \"dec9abf1-8b69-46d3-b5da-cb0b552b8447\") " pod="kuadrant-system/authorino-cbbff8c7f-crdpg" Nov 04 03:57:32.825886 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:32.825715 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dec9abf1-8b69-46d3-b5da-cb0b552b8447-tls-cert\") pod \"authorino-cbbff8c7f-crdpg\" (UID: \"dec9abf1-8b69-46d3-b5da-cb0b552b8447\") " pod="kuadrant-system/authorino-cbbff8c7f-crdpg" Nov 04 03:57:32.926368 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:32.926285 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnlk8\" (UniqueName: \"kubernetes.io/projected/dec9abf1-8b69-46d3-b5da-cb0b552b8447-kube-api-access-qnlk8\") pod \"authorino-cbbff8c7f-crdpg\" (UID: \"dec9abf1-8b69-46d3-b5da-cb0b552b8447\") " pod="kuadrant-system/authorino-cbbff8c7f-crdpg" Nov 04 03:57:32.926368 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:32.926337 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dec9abf1-8b69-46d3-b5da-cb0b552b8447-tls-cert\") pod \"authorino-cbbff8c7f-crdpg\" (UID: \"dec9abf1-8b69-46d3-b5da-cb0b552b8447\") " pod="kuadrant-system/authorino-cbbff8c7f-crdpg" Nov 04 03:57:32.928763 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:32.928742 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dec9abf1-8b69-46d3-b5da-cb0b552b8447-tls-cert\") pod \"authorino-cbbff8c7f-crdpg\" (UID: \"dec9abf1-8b69-46d3-b5da-cb0b552b8447\") " pod="kuadrant-system/authorino-cbbff8c7f-crdpg" Nov 04 03:57:32.939091 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:32.939064 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnlk8\" (UniqueName: \"kubernetes.io/projected/dec9abf1-8b69-46d3-b5da-cb0b552b8447-kube-api-access-qnlk8\") pod \"authorino-cbbff8c7f-crdpg\" (UID: \"dec9abf1-8b69-46d3-b5da-cb0b552b8447\") " pod="kuadrant-system/authorino-cbbff8c7f-crdpg" Nov 04 03:57:33.002065 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:33.002021 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cbbff8c7f-crdpg" Nov 04 03:57:33.291445 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:33.291419 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cbbff8c7f-crdpg"] Nov 04 03:57:33.293971 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:57:33.293940 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddec9abf1_8b69_46d3_b5da_cb0b552b8447.slice/crio-15080e062b4c0ab609a8dff60b9ff65d5212a8e87406f4d497f6fc4d61353600 WatchSource:0}: Error finding container 15080e062b4c0ab609a8dff60b9ff65d5212a8e87406f4d497f6fc4d61353600: Status 404 returned error can't find the container with id 15080e062b4c0ab609a8dff60b9ff65d5212a8e87406f4d497f6fc4d61353600 Nov 04 03:57:33.377943 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:33.374263 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 04 03:57:33.377943 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:33.374336 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" podUID="177f8aff-8b11-4891-9648-9c7aeb9305b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 04 03:57:33.912900 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:33.912861 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cbbff8c7f-crdpg" event={"ID":"dec9abf1-8b69-46d3-b5da-cb0b552b8447","Type":"ContainerStarted","Data":"15080e062b4c0ab609a8dff60b9ff65d5212a8e87406f4d497f6fc4d61353600"} Nov 04 03:57:34.366031 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:34.366000 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 04 03:57:34.366192 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:34.366082 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" podUID="177f8aff-8b11-4891-9648-9c7aeb9305b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 04 03:57:34.921059 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:34.921012 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cbbff8c7f-crdpg" event={"ID":"dec9abf1-8b69-46d3-b5da-cb0b552b8447","Type":"ContainerStarted","Data":"7461a5d82000f95615f449aa0122210d2d4aaae07aa62a2ae4788476233f6640"} Nov 04 03:57:34.975673 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:34.975603 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-cbbff8c7f-crdpg" podStartSLOduration=2.033737151 podStartE2EDuration="2.97558399s" totalImagesPullingTime="941.846839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:57:34.974266794 +0000 UTC m=+498.775212602" watchObservedRunningTime="2026-11-04 03:57:34.97558399 +0000 UTC m=+498.776529799" Nov 04 03:57:35.366102 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:35.366054 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 04 03:57:35.366283 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:35.366132 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" podUID="177f8aff-8b11-4891-9648-9c7aeb9305b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 04 03:57:35.707002 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:57:35.706897 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:57:36.418655 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:36.418617 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:36.419886 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:36.419863 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:36.420600 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:36.420584 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8" Nov 04 03:57:42.133241 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.133199 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92"] Nov 04 03:57:42.140520 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.140489 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.157162 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.157136 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92"] Nov 04 03:57:42.217628 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.217592 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa513df7-058f-4dbc-8743-f0b1e3f5e088-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.217775 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.217650 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa513df7-058f-4dbc-8743-f0b1e3f5e088-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.217775 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.217736 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa513df7-058f-4dbc-8743-f0b1e3f5e088-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.217854 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.217799 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7v2k\" (UniqueName: \"kubernetes.io/projected/aa513df7-058f-4dbc-8743-f0b1e3f5e088-kube-api-access-z7v2k\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.217887 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.217861 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa513df7-058f-4dbc-8743-f0b1e3f5e088-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.217943 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.217903 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa513df7-058f-4dbc-8743-f0b1e3f5e088-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.217979 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.217944 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa513df7-058f-4dbc-8743-f0b1e3f5e088-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.218043 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.218023 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa513df7-058f-4dbc-8743-f0b1e3f5e088-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.218095 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.218083 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa513df7-058f-4dbc-8743-f0b1e3f5e088-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.319524 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.319492 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7v2k\" (UniqueName: \"kubernetes.io/projected/aa513df7-058f-4dbc-8743-f0b1e3f5e088-kube-api-access-z7v2k\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.319684 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.319536 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa513df7-058f-4dbc-8743-f0b1e3f5e088-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.319684 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.319561 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa513df7-058f-4dbc-8743-f0b1e3f5e088-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.319684 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.319583 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa513df7-058f-4dbc-8743-f0b1e3f5e088-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.319684 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.319616 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa513df7-058f-4dbc-8743-f0b1e3f5e088-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.319684 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.319639 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa513df7-058f-4dbc-8743-f0b1e3f5e088-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.319684 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.319684 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa513df7-058f-4dbc-8743-f0b1e3f5e088-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.320040 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.319706 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa513df7-058f-4dbc-8743-f0b1e3f5e088-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.320040 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.319735 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa513df7-058f-4dbc-8743-f0b1e3f5e088-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.320040 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.320021 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa513df7-058f-4dbc-8743-f0b1e3f5e088-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.320168 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.320079 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa513df7-058f-4dbc-8743-f0b1e3f5e088-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.320203 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.320178 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa513df7-058f-4dbc-8743-f0b1e3f5e088-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.320268 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.320245 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa513df7-058f-4dbc-8743-f0b1e3f5e088-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.320433 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.320410 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa513df7-058f-4dbc-8743-f0b1e3f5e088-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.321909 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.321886 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa513df7-058f-4dbc-8743-f0b1e3f5e088-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.322262 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.322243 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa513df7-058f-4dbc-8743-f0b1e3f5e088-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.337146 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.337118 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7v2k\" (UniqueName: \"kubernetes.io/projected/aa513df7-058f-4dbc-8743-f0b1e3f5e088-kube-api-access-z7v2k\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.337252 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.337143 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa513df7-058f-4dbc-8743-f0b1e3f5e088-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-n5x92\" (UID: \"aa513df7-058f-4dbc-8743-f0b1e3f5e088\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.454490 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.454398 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:42.787862 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.787835 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92"] Nov 04 03:57:42.789658 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:57:42.789631 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa513df7_058f_4dbc_8743_f0b1e3f5e088.slice/crio-34be287141230c1ad050e80fb9e74652d05e86a64feaeebd9441f5385fe9a1c7 WatchSource:0}: Error finding container 34be287141230c1ad050e80fb9e74652d05e86a64feaeebd9441f5385fe9a1c7: Status 404 returned error can't find the container with id 34be287141230c1ad050e80fb9e74652d05e86a64feaeebd9441f5385fe9a1c7 Nov 04 03:57:42.791964 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.791907 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 03:57:42.792037 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.791996 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 03:57:42.792037 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.792028 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 03:57:42.965105 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.965067 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" event={"ID":"aa513df7-058f-4dbc-8743-f0b1e3f5e088","Type":"ContainerStarted","Data":"df82e646558f0848c30e216c578a57486a8846b15bd65e9ed7baf216810c2c5f"} Nov 04 03:57:42.965298 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:42.965129 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" event={"ID":"aa513df7-058f-4dbc-8743-f0b1e3f5e088","Type":"ContainerStarted","Data":"34be287141230c1ad050e80fb9e74652d05e86a64feaeebd9441f5385fe9a1c7"} Nov 04 03:57:43.000943 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.000868 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" podStartSLOduration=1.000852621 podStartE2EDuration="1.000852621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:57:42.99859624 +0000 UTC m=+506.799542047" watchObservedRunningTime="2026-11-04 03:57:43.000852621 +0000 UTC m=+506.801798429" Nov 04 03:57:43.004278 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.004252 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-lqlbm"] Nov 04 03:57:43.008143 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.008126 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lqlbm" Nov 04 03:57:43.017540 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.017502 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 04 03:57:43.017540 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.017502 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-42t87\"" Nov 04 03:57:43.021284 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.021260 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-lqlbm"] Nov 04 03:57:43.030427 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.030403 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-sv8xh"] Nov 04 03:57:43.034515 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.034491 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-sv8xh" Nov 04 03:57:43.036442 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.036408 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94562f01-24b5-4df4-ae94-967d5a116a9c-plugins-config-volume\") pod \"payload-processing-7bb9489676-sv8xh\" (UID: \"94562f01-24b5-4df4-ae94-967d5a116a9c\") " pod="openshift-ingress/payload-processing-7bb9489676-sv8xh" Nov 04 03:57:43.036596 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.036446 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94j5p\" (UniqueName: \"kubernetes.io/projected/94562f01-24b5-4df4-ae94-967d5a116a9c-kube-api-access-94j5p\") pod \"payload-processing-7bb9489676-sv8xh\" (UID: \"94562f01-24b5-4df4-ae94-967d5a116a9c\") " pod="openshift-ingress/payload-processing-7bb9489676-sv8xh" Nov 04 03:57:43.036596 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.036543 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ea3aa4-a31f-430e-ab69-31d81f975218-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-lqlbm\" (UID: \"c5ea3aa4-a31f-430e-ab69-31d81f975218\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lqlbm" Nov 04 03:57:43.036730 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.036688 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n2ht\" (UniqueName: \"kubernetes.io/projected/c5ea3aa4-a31f-430e-ab69-31d81f975218-kube-api-access-9n2ht\") pod \"payload-pre-processing-7b98bb678c-lqlbm\" (UID: \"c5ea3aa4-a31f-430e-ab69-31d81f975218\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lqlbm" Nov 04 03:57:43.045455 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.045432 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-sv8xh"] Nov 04 03:57:43.137448 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.137404 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ea3aa4-a31f-430e-ab69-31d81f975218-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-lqlbm\" (UID: \"c5ea3aa4-a31f-430e-ab69-31d81f975218\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lqlbm" Nov 04 03:57:43.137880 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.137514 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n2ht\" (UniqueName: \"kubernetes.io/projected/c5ea3aa4-a31f-430e-ab69-31d81f975218-kube-api-access-9n2ht\") pod \"payload-pre-processing-7b98bb678c-lqlbm\" (UID: \"c5ea3aa4-a31f-430e-ab69-31d81f975218\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lqlbm" Nov 04 03:57:43.137880 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.137602 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94562f01-24b5-4df4-ae94-967d5a116a9c-plugins-config-volume\") pod \"payload-processing-7bb9489676-sv8xh\" (UID: \"94562f01-24b5-4df4-ae94-967d5a116a9c\") " pod="openshift-ingress/payload-processing-7bb9489676-sv8xh" Nov 04 03:57:43.137880 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.137629 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94j5p\" (UniqueName: \"kubernetes.io/projected/94562f01-24b5-4df4-ae94-967d5a116a9c-kube-api-access-94j5p\") pod \"payload-processing-7bb9489676-sv8xh\" (UID: \"94562f01-24b5-4df4-ae94-967d5a116a9c\") " pod="openshift-ingress/payload-processing-7bb9489676-sv8xh" Nov 04 03:57:43.138166 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.138144 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ea3aa4-a31f-430e-ab69-31d81f975218-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-lqlbm\" (UID: \"c5ea3aa4-a31f-430e-ab69-31d81f975218\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lqlbm" Nov 04 03:57:43.138479 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.138454 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94562f01-24b5-4df4-ae94-967d5a116a9c-plugins-config-volume\") pod \"payload-processing-7bb9489676-sv8xh\" (UID: \"94562f01-24b5-4df4-ae94-967d5a116a9c\") " pod="openshift-ingress/payload-processing-7bb9489676-sv8xh" Nov 04 03:57:43.154203 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.154102 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n2ht\" (UniqueName: \"kubernetes.io/projected/c5ea3aa4-a31f-430e-ab69-31d81f975218-kube-api-access-9n2ht\") pod \"payload-pre-processing-7b98bb678c-lqlbm\" (UID: \"c5ea3aa4-a31f-430e-ab69-31d81f975218\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lqlbm" Nov 04 03:57:43.154203 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.154127 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94j5p\" (UniqueName: \"kubernetes.io/projected/94562f01-24b5-4df4-ae94-967d5a116a9c-kube-api-access-94j5p\") pod \"payload-processing-7bb9489676-sv8xh\" (UID: \"94562f01-24b5-4df4-ae94-967d5a116a9c\") " pod="openshift-ingress/payload-processing-7bb9489676-sv8xh" Nov 04 03:57:43.323738 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.323700 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lqlbm" Nov 04 03:57:43.370954 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.370142 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-sv8xh" Nov 04 03:57:43.479447 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.464179 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:43.480481 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.480127 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 03:57:43.480481 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.480184 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" podUID="aa513df7-058f-4dbc-8743-f0b1e3f5e088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 03:57:43.692679 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.692644 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-lqlbm"] Nov 04 03:57:43.740843 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.740813 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-sv8xh"] Nov 04 03:57:43.742326 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:57:43.742305 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94562f01_24b5_4df4_ae94_967d5a116a9c.slice/crio-1d27377d8db8881888d79ccb45aa1b756adaad44979ac1c95a09365c528da7fc WatchSource:0}: Error finding container 1d27377d8db8881888d79ccb45aa1b756adaad44979ac1c95a09365c528da7fc: Status 404 returned error can't find the container with id 1d27377d8db8881888d79ccb45aa1b756adaad44979ac1c95a09365c528da7fc Nov 04 03:57:43.999315 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:43.999225 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-sv8xh" event={"ID":"94562f01-24b5-4df4-ae94-967d5a116a9c","Type":"ContainerStarted","Data":"1d27377d8db8881888d79ccb45aa1b756adaad44979ac1c95a09365c528da7fc"} Nov 04 03:57:44.000742 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:44.000700 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lqlbm" event={"ID":"c5ea3aa4-a31f-430e-ab69-31d81f975218","Type":"ContainerStarted","Data":"31ca8b5ed75e4bdc611c98e55c73d441261199de32486c38a16e451889474107"} Nov 04 03:57:44.486943 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:44.485027 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 03:57:44.486943 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:44.485096 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" podUID="aa513df7-058f-4dbc-8743-f0b1e3f5e088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 03:57:45.472829 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:45.472350 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 03:57:45.472829 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:45.472416 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" podUID="aa513df7-058f-4dbc-8743-f0b1e3f5e088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 03:57:46.478801 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:46.478300 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 03:57:46.478801 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:46.478365 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" podUID="aa513df7-058f-4dbc-8743-f0b1e3f5e088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 03:57:47.476958 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:47.468978 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 03:57:47.476958 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:47.469042 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" podUID="aa513df7-058f-4dbc-8743-f0b1e3f5e088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 03:57:48.029561 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:48.029521 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-sv8xh" event={"ID":"94562f01-24b5-4df4-ae94-967d5a116a9c","Type":"ContainerStarted","Data":"632269c6116ccffdc8a56cbb6775f6cee0ce38155945a01cb3a1005da0ab75d4"} Nov 04 03:57:48.029994 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:48.029678 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-sv8xh" Nov 04 03:57:48.031337 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:48.031310 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lqlbm" event={"ID":"c5ea3aa4-a31f-430e-ab69-31d81f975218","Type":"ContainerStarted","Data":"f5557bf1c48ef64c5b3e205d42357aa4de08b1aa547af10d73fa44b0908e6321"} Nov 04 03:57:48.031442 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:48.031389 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lqlbm" Nov 04 03:57:48.050998 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:48.050956 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-sv8xh" podStartSLOduration=1.2061305230000001 podStartE2EDuration="5.05094291s" totalImagesPullingTime="3.844812387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:57:48.048878349 +0000 UTC m=+511.849824160" watchObservedRunningTime="2026-11-04 03:57:48.05094291 +0000 UTC m=+511.851888717" Nov 04 03:57:48.079071 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:48.079015 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lqlbm" podStartSLOduration=2.244545086 podStartE2EDuration="6.078997591s" totalImagesPullingTime="3.834452505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:57:48.078316847 +0000 UTC m=+511.879262648" watchObservedRunningTime="2026-11-04 03:57:48.078997591 +0000 UTC m=+511.879943399" Nov 04 03:57:48.466945 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:48.457325 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 03:57:48.466945 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:48.457394 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" podUID="aa513df7-058f-4dbc-8743-f0b1e3f5e088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 03:57:49.455657 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:49.455609 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 03:57:49.456102 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:49.455724 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" podUID="aa513df7-058f-4dbc-8743-f0b1e3f5e088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 03:57:50.455394 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:50.455358 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 03:57:50.455577 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:50.455414 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" podUID="aa513df7-058f-4dbc-8743-f0b1e3f5e088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 03:57:52.497138 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:52.497096 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:57:52.497559 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:52.497174 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" podUID="aa513df7-058f-4dbc-8743-f0b1e3f5e088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:57:53.538174 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:53.538133 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:57:53.538619 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:53.538202 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" podUID="aa513df7-058f-4dbc-8743-f0b1e3f5e088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:57:53.765287 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:53.765239 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:53.765676 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:53.765646 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:53.767139 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:53.767106 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92" Nov 04 03:57:59.040524 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:59.040490 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-sv8xh" Nov 04 03:57:59.040992 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:57:59.040562 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lqlbm" Nov 04 03:58:05.678221 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:58:05.678177 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:58:35.678153 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:58:35.678027 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:58:39.648168 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.648132 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs"] Nov 04 03:58:39.651948 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.651905 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs" Nov 04 03:58:39.666445 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.666414 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs"] Nov 04 03:58:39.674471 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.674447 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-998997\"" Nov 04 03:58:39.674592 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.674504 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-998997-dockercfg-28zdh\"" Nov 04 03:58:39.679535 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.679512 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn"] Nov 04 03:58:39.683545 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.683523 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn" Nov 04 03:58:39.699327 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.699300 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn"] Nov 04 03:58:39.758068 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.758028 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08fb1e7a-0bdb-4529-a421-345600882d98-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs\" (UID: \"08fb1e7a-0bdb-4529-a421-345600882d98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs" Nov 04 03:58:39.758375 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.758350 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tvqt\" (UniqueName: \"kubernetes.io/projected/08fb1e7a-0bdb-4529-a421-345600882d98-kube-api-access-2tvqt\") pod \"payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs\" (UID: \"08fb1e7a-0bdb-4529-a421-345600882d98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs" Nov 04 03:58:39.859237 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.859196 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c9dc55-1b5a-4bcc-94de-7b3eaa030b62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn\" (UID: \"61c9dc55-1b5a-4bcc-94de-7b3eaa030b62\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn" Nov 04 03:58:39.859440 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.859259 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08fb1e7a-0bdb-4529-a421-345600882d98-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs\" (UID: \"08fb1e7a-0bdb-4529-a421-345600882d98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs" Nov 04 03:58:39.859440 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.859381 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tvqt\" (UniqueName: \"kubernetes.io/projected/08fb1e7a-0bdb-4529-a421-345600882d98-kube-api-access-2tvqt\") pod \"payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs\" (UID: \"08fb1e7a-0bdb-4529-a421-345600882d98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs" Nov 04 03:58:39.859639 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.859509 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2q6p\" (UniqueName: \"kubernetes.io/projected/61c9dc55-1b5a-4bcc-94de-7b3eaa030b62-kube-api-access-m2q6p\") pod \"payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn\" (UID: \"61c9dc55-1b5a-4bcc-94de-7b3eaa030b62\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn" Nov 04 03:58:39.860682 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.860654 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08fb1e7a-0bdb-4529-a421-345600882d98-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs\" (UID: \"08fb1e7a-0bdb-4529-a421-345600882d98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs" Nov 04 03:58:39.894981 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.894944 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tvqt\" (UniqueName: \"kubernetes.io/projected/08fb1e7a-0bdb-4529-a421-345600882d98-kube-api-access-2tvqt\") pod \"payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs\" (UID: \"08fb1e7a-0bdb-4529-a421-345600882d98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs" Nov 04 03:58:39.960881 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.960803 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2q6p\" (UniqueName: \"kubernetes.io/projected/61c9dc55-1b5a-4bcc-94de-7b3eaa030b62-kube-api-access-m2q6p\") pod \"payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn\" (UID: \"61c9dc55-1b5a-4bcc-94de-7b3eaa030b62\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn" Nov 04 03:58:39.960881 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.960868 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c9dc55-1b5a-4bcc-94de-7b3eaa030b62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn\" (UID: \"61c9dc55-1b5a-4bcc-94de-7b3eaa030b62\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn" Nov 04 03:58:39.961572 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.961551 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c9dc55-1b5a-4bcc-94de-7b3eaa030b62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn\" (UID: \"61c9dc55-1b5a-4bcc-94de-7b3eaa030b62\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn" Nov 04 03:58:39.966663 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:39.966639 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs" Nov 04 03:58:40.019583 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:40.019518 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2q6p\" (UniqueName: \"kubernetes.io/projected/61c9dc55-1b5a-4bcc-94de-7b3eaa030b62-kube-api-access-m2q6p\") pod \"payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn\" (UID: \"61c9dc55-1b5a-4bcc-94de-7b3eaa030b62\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn" Nov 04 03:58:40.301863 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:40.301788 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn" Nov 04 03:58:40.328831 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:40.328792 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs"] Nov 04 03:58:40.330900 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:58:40.330859 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08fb1e7a_0bdb_4529_a421_345600882d98.slice/crio-182de54b6d81ec2280a7fd15b98e0c17ddedbe1fce446423ff83b1742a21f3d9 WatchSource:0}: Error finding container 182de54b6d81ec2280a7fd15b98e0c17ddedbe1fce446423ff83b1742a21f3d9: Status 404 returned error can't find the container with id 182de54b6d81ec2280a7fd15b98e0c17ddedbe1fce446423ff83b1742a21f3d9 Nov 04 03:58:40.601999 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:40.601969 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn"] Nov 04 03:58:40.604679 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:58:40.604641 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c9dc55_1b5a_4bcc_94de_7b3eaa030b62.slice/crio-c13d967f30ac047b9deeb948e450c4e6b88bbaeb0ef98b5c76061548348d1572 WatchSource:0}: Error finding container c13d967f30ac047b9deeb948e450c4e6b88bbaeb0ef98b5c76061548348d1572: Status 404 returned error can't find the container with id c13d967f30ac047b9deeb948e450c4e6b88bbaeb0ef98b5c76061548348d1572 Nov 04 03:58:41.207509 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.207473 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st"] Nov 04 03:58:41.211360 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.211336 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" Nov 04 03:58:41.223989 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.223942 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st"] Nov 04 03:58:41.225430 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.225405 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-4afe0e-dockercfg-jrqdr\"" Nov 04 03:58:41.225593 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.225405 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-4afe0e\"" Nov 04 03:58:41.273639 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.273601 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6"] Nov 04 03:58:41.279395 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.279367 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62"] Nov 04 03:58:41.279559 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.279492 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6" Nov 04 03:58:41.283205 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.283180 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" Nov 04 03:58:41.305955 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.294331 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6"] Nov 04 03:58:41.305955 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.298890 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62"] Nov 04 03:58:41.324651 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.324621 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-998997-serving-cert\"" Nov 04 03:58:41.331211 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.331167 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn" event={"ID":"61c9dc55-1b5a-4bcc-94de-7b3eaa030b62","Type":"ContainerStarted","Data":"c07477dbb920bf4e1bee79da495c060dc6ef12482f968bea5964b72f7c109e01"} Nov 04 03:58:41.331408 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.331233 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn" Nov 04 03:58:41.331408 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.331254 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn" event={"ID":"61c9dc55-1b5a-4bcc-94de-7b3eaa030b62","Type":"ContainerStarted","Data":"c13d967f30ac047b9deeb948e450c4e6b88bbaeb0ef98b5c76061548348d1572"} Nov 04 03:58:41.332955 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.332898 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs" event={"ID":"08fb1e7a-0bdb-4529-a421-345600882d98","Type":"ContainerStarted","Data":"166c42e26fbc231f139521ade0a1cfc6b3aa9588c8d4a7c712fe5297d3658fb7"} Nov 04 03:58:41.333100 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.332972 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs" event={"ID":"08fb1e7a-0bdb-4529-a421-345600882d98","Type":"ContainerStarted","Data":"182de54b6d81ec2280a7fd15b98e0c17ddedbe1fce446423ff83b1742a21f3d9"} Nov 04 03:58:41.333100 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.333029 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs" Nov 04 03:58:41.362649 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.362589 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs" podStartSLOduration=2.362570734 podStartE2EDuration="2.362570734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:58:41.359161781 +0000 UTC m=+565.160107602" watchObservedRunningTime="2026-11-04 03:58:41.362570734 +0000 UTC m=+565.163516544" Nov 04 03:58:41.377805 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.377771 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzgw2\" (UniqueName: \"kubernetes.io/projected/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-kube-api-access-tzgw2\") pod \"maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62\" (UID: \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" Nov 04 03:58:41.377982 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.377850 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62\" (UID: \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" Nov 04 03:58:41.377982 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.377936 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-metrics-certs\") pod \"maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62\" (UID: \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" Nov 04 03:58:41.378124 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.378032 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkk5p\" (UniqueName: \"kubernetes.io/projected/c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1-kube-api-access-kkk5p\") pod \"payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st\" (UID: \"c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" Nov 04 03:58:41.378124 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.378052 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6626886d-db18-4cd8-ae04-23e2b5841223-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6\" (UID: \"6626886d-db18-4cd8-ae04-23e2b5841223\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6" Nov 04 03:58:41.378228 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.378129 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st\" (UID: \"c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" Nov 04 03:58:41.378228 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.378156 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62\" (UID: \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" Nov 04 03:58:41.378384 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.378247 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h56g9\" (UniqueName: \"kubernetes.io/projected/6626886d-db18-4cd8-ae04-23e2b5841223-kube-api-access-h56g9\") pod \"payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6\" (UID: \"6626886d-db18-4cd8-ae04-23e2b5841223\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6" Nov 04 03:58:41.385211 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.385165 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn" podStartSLOduration=2.385149223 podStartE2EDuration="2.385149223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:58:41.382211742 +0000 UTC m=+565.183157550" watchObservedRunningTime="2026-11-04 03:58:41.385149223 +0000 UTC m=+565.186095032" Nov 04 03:58:41.479641 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.479514 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkk5p\" (UniqueName: \"kubernetes.io/projected/c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1-kube-api-access-kkk5p\") pod \"payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st\" (UID: \"c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" Nov 04 03:58:41.479641 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.479596 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6626886d-db18-4cd8-ae04-23e2b5841223-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6\" (UID: \"6626886d-db18-4cd8-ae04-23e2b5841223\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6" Nov 04 03:58:41.479881 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.479639 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st\" (UID: \"c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" Nov 04 03:58:41.479881 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.479679 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62\" (UID: \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" Nov 04 03:58:41.479881 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.479803 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h56g9\" (UniqueName: \"kubernetes.io/projected/6626886d-db18-4cd8-ae04-23e2b5841223-kube-api-access-h56g9\") pod \"payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6\" (UID: \"6626886d-db18-4cd8-ae04-23e2b5841223\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6" Nov 04 03:58:41.480091 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.479910 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzgw2\" (UniqueName: \"kubernetes.io/projected/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-kube-api-access-tzgw2\") pod \"maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62\" (UID: \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" Nov 04 03:58:41.480091 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.479997 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62\" (UID: \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" Nov 04 03:58:41.480091 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.480059 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-metrics-certs\") pod \"maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62\" (UID: \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" Nov 04 03:58:41.480785 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.480587 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6626886d-db18-4cd8-ae04-23e2b5841223-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6\" (UID: \"6626886d-db18-4cd8-ae04-23e2b5841223\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6" Nov 04 03:58:41.480944 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.480823 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st\" (UID: \"c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" Nov 04 03:58:41.481090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.481067 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62\" (UID: \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" Nov 04 03:58:41.482881 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.482862 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-metrics-certs\") pod \"maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62\" (UID: \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" Nov 04 03:58:41.483057 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.483034 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62\" (UID: \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" Nov 04 03:58:41.492191 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.492158 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h56g9\" (UniqueName: \"kubernetes.io/projected/6626886d-db18-4cd8-ae04-23e2b5841223-kube-api-access-h56g9\") pod \"payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6\" (UID: \"6626886d-db18-4cd8-ae04-23e2b5841223\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6" Nov 04 03:58:41.492526 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.492508 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzgw2\" (UniqueName: \"kubernetes.io/projected/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-kube-api-access-tzgw2\") pod \"maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62\" (UID: \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" Nov 04 03:58:41.492755 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.492707 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkk5p\" (UniqueName: \"kubernetes.io/projected/c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1-kube-api-access-kkk5p\") pod \"payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st\" (UID: \"c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" Nov 04 03:58:41.527269 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.527238 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" Nov 04 03:58:41.609343 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.609292 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6" Nov 04 03:58:41.612087 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.612058 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" Nov 04 03:58:41.834181 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.834145 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st"] Nov 04 03:58:41.834795 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:58:41.834766 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7920bcb_2a60_45c0_bc3d_0d28ae3fc3d1.slice/crio-7b250f1effe41e03e2644980fe184778e43af5da537a8009cb424377f47b4ca1 WatchSource:0}: Error finding container 7b250f1effe41e03e2644980fe184778e43af5da537a8009cb424377f47b4ca1: Status 404 returned error can't find the container with id 7b250f1effe41e03e2644980fe184778e43af5da537a8009cb424377f47b4ca1 Nov 04 03:58:41.900489 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.900410 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6"] Nov 04 03:58:41.910198 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:58:41.910151 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6626886d_db18_4cd8_ae04_23e2b5841223.slice/crio-0396385859e38a83f25989136938461400fb785d88c3abfa7b940b129b0629a9 WatchSource:0}: Error finding container 0396385859e38a83f25989136938461400fb785d88c3abfa7b940b129b0629a9: Status 404 returned error can't find the container with id 0396385859e38a83f25989136938461400fb785d88c3abfa7b940b129b0629a9 Nov 04 03:58:41.916836 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:41.916810 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62"] Nov 04 03:58:41.918432 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:58:41.918406 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3324a6f_e5f3_4ea0_9e43_e2900ae045f1.slice/crio-6f04410bdbfd319b6628d2745ae17811dadaa43f024f0fa4e25b5c7c84400ad5 WatchSource:0}: Error finding container 6f04410bdbfd319b6628d2745ae17811dadaa43f024f0fa4e25b5c7c84400ad5: Status 404 returned error can't find the container with id 6f04410bdbfd319b6628d2745ae17811dadaa43f024f0fa4e25b5c7c84400ad5 Nov 04 03:58:42.341377 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.341339 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" event={"ID":"c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1","Type":"ContainerStarted","Data":"211e4acc080cca8f065a7480f31f1cacbd7d72d478286d335b9588a69e33c5ba"} Nov 04 03:58:42.341950 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.341396 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" event={"ID":"c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1","Type":"ContainerStarted","Data":"7b250f1effe41e03e2644980fe184778e43af5da537a8009cb424377f47b4ca1"} Nov 04 03:58:42.341950 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.341425 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" Nov 04 03:58:42.343246 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.343020 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6" event={"ID":"6626886d-db18-4cd8-ae04-23e2b5841223","Type":"ContainerStarted","Data":"5229455eca23d4b2d5b8da2f671625a6ac1064d75932fdbd28f7f26ccf15e285"} Nov 04 03:58:42.343246 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.343077 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6" event={"ID":"6626886d-db18-4cd8-ae04-23e2b5841223","Type":"ContainerStarted","Data":"0396385859e38a83f25989136938461400fb785d88c3abfa7b940b129b0629a9"} Nov 04 03:58:42.343246 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.343107 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6" Nov 04 03:58:42.344507 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.344481 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" event={"ID":"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1","Type":"ContainerStarted","Data":"6f04410bdbfd319b6628d2745ae17811dadaa43f024f0fa4e25b5c7c84400ad5"} Nov 04 03:58:42.345250 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.345227 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp"] Nov 04 03:58:42.349166 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.349150 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" Nov 04 03:58:42.369322 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.369295 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-56a44f-dockercfg-7m7xn\"" Nov 04 03:58:42.371967 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.371942 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-56a44f\"" Nov 04 03:58:42.379210 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.379183 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp"] Nov 04 03:58:42.381541 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.381498 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" podStartSLOduration=1.3814851 podStartE2EDuration="1.3814851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:58:42.37707392 +0000 UTC m=+566.178019729" watchObservedRunningTime="2026-11-04 03:58:42.3814851 +0000 UTC m=+566.182430909" Nov 04 03:58:42.399909 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.399870 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp"] Nov 04 03:58:42.404314 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.404293 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" Nov 04 03:58:42.451680 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.451333 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6" podStartSLOduration=1.451319381 podStartE2EDuration="1.451319381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:58:42.44800483 +0000 UTC m=+566.248950638" watchObservedRunningTime="2026-11-04 03:58:42.451319381 +0000 UTC m=+566.252265224" Nov 04 03:58:42.453308 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.453276 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp"] Nov 04 03:58:42.463647 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.463619 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-4afe0e-serving-cert\"" Nov 04 03:58:42.496027 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.496003 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff721376-eba2-4a06-b1b9-48e5875d9c05-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp\" (UID: \"ff721376-eba2-4a06-b1b9-48e5875d9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" Nov 04 03:58:42.496136 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.496096 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sg5d\" (UniqueName: \"kubernetes.io/projected/ff721376-eba2-4a06-b1b9-48e5875d9c05-kube-api-access-7sg5d\") pod \"maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp\" (UID: \"ff721376-eba2-4a06-b1b9-48e5875d9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" Nov 04 03:58:42.496228 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.496193 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svn79\" (UniqueName: \"kubernetes.io/projected/28e01b06-4f41-4ec9-b351-6145f5d0bd2a-kube-api-access-svn79\") pod \"payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp\" (UID: \"28e01b06-4f41-4ec9-b351-6145f5d0bd2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" Nov 04 03:58:42.496466 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.496439 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e01b06-4f41-4ec9-b351-6145f5d0bd2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp\" (UID: \"28e01b06-4f41-4ec9-b351-6145f5d0bd2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" Nov 04 03:58:42.496571 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.496501 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff721376-eba2-4a06-b1b9-48e5875d9c05-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp\" (UID: \"ff721376-eba2-4a06-b1b9-48e5875d9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" Nov 04 03:58:42.496571 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.496535 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff721376-eba2-4a06-b1b9-48e5875d9c05-metrics-certs\") pod \"maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp\" (UID: \"ff721376-eba2-4a06-b1b9-48e5875d9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" Nov 04 03:58:42.597554 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.597460 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff721376-eba2-4a06-b1b9-48e5875d9c05-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp\" (UID: \"ff721376-eba2-4a06-b1b9-48e5875d9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" Nov 04 03:58:42.597554 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.597511 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff721376-eba2-4a06-b1b9-48e5875d9c05-metrics-certs\") pod \"maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp\" (UID: \"ff721376-eba2-4a06-b1b9-48e5875d9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" Nov 04 03:58:42.597779 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.597637 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff721376-eba2-4a06-b1b9-48e5875d9c05-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp\" (UID: \"ff721376-eba2-4a06-b1b9-48e5875d9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" Nov 04 03:58:42.597779 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.597707 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sg5d\" (UniqueName: \"kubernetes.io/projected/ff721376-eba2-4a06-b1b9-48e5875d9c05-kube-api-access-7sg5d\") pod \"maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp\" (UID: \"ff721376-eba2-4a06-b1b9-48e5875d9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" Nov 04 03:58:42.597779 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.597745 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svn79\" (UniqueName: \"kubernetes.io/projected/28e01b06-4f41-4ec9-b351-6145f5d0bd2a-kube-api-access-svn79\") pod \"payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp\" (UID: \"28e01b06-4f41-4ec9-b351-6145f5d0bd2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" Nov 04 03:58:42.598016 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.597780 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e01b06-4f41-4ec9-b351-6145f5d0bd2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp\" (UID: \"28e01b06-4f41-4ec9-b351-6145f5d0bd2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" Nov 04 03:58:42.598016 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:58:42.597795 4966 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-serving-cert: secret "maas-api-e2e-worker-w0-4afe0e-serving-cert" not found Nov 04 03:58:42.598016 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:58:42.597870 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff721376-eba2-4a06-b1b9-48e5875d9c05-maas-api-tls podName:ff721376-eba2-4a06-b1b9-48e5875d9c05 nodeName:}" failed. No retries permitted until 2026-11-04 03:58:43.097849908 +0000 UTC m=+566.898795700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff721376-eba2-4a06-b1b9-48e5875d9c05-maas-api-tls") pod "maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" (UID: "ff721376-eba2-4a06-b1b9-48e5875d9c05") : secret "maas-api-e2e-worker-w0-4afe0e-serving-cert" not found Nov 04 03:58:42.598281 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.598258 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff721376-eba2-4a06-b1b9-48e5875d9c05-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp\" (UID: \"ff721376-eba2-4a06-b1b9-48e5875d9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" Nov 04 03:58:42.598587 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.598565 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e01b06-4f41-4ec9-b351-6145f5d0bd2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp\" (UID: \"28e01b06-4f41-4ec9-b351-6145f5d0bd2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" Nov 04 03:58:42.600273 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.600249 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff721376-eba2-4a06-b1b9-48e5875d9c05-metrics-certs\") pod \"maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp\" (UID: \"ff721376-eba2-4a06-b1b9-48e5875d9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" Nov 04 03:58:42.611456 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.611427 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sg5d\" (UniqueName: \"kubernetes.io/projected/ff721376-eba2-4a06-b1b9-48e5875d9c05-kube-api-access-7sg5d\") pod \"maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp\" (UID: \"ff721376-eba2-4a06-b1b9-48e5875d9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" Nov 04 03:58:42.613016 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.612988 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svn79\" (UniqueName: \"kubernetes.io/projected/28e01b06-4f41-4ec9-b351-6145f5d0bd2a-kube-api-access-svn79\") pod \"payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp\" (UID: \"28e01b06-4f41-4ec9-b351-6145f5d0bd2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" Nov 04 03:58:42.662362 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.662314 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" Nov 04 03:58:42.950102 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:42.950066 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp"] Nov 04 03:58:42.951141 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:58:42.951107 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e01b06_4f41_4ec9_b351_6145f5d0bd2a.slice/crio-7254bd2a02bee8fb4c6e3360cae55e8dbbffd2116845bd33b8119dad733309fb WatchSource:0}: Error finding container 7254bd2a02bee8fb4c6e3360cae55e8dbbffd2116845bd33b8119dad733309fb: Status 404 returned error can't find the container with id 7254bd2a02bee8fb4c6e3360cae55e8dbbffd2116845bd33b8119dad733309fb Nov 04 03:58:43.103775 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.103736 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff721376-eba2-4a06-b1b9-48e5875d9c05-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp\" (UID: \"ff721376-eba2-4a06-b1b9-48e5875d9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" Nov 04 03:58:43.106524 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.106497 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff721376-eba2-4a06-b1b9-48e5875d9c05-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp\" (UID: \"ff721376-eba2-4a06-b1b9-48e5875d9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" Nov 04 03:58:43.309950 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.309904 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2"] Nov 04 03:58:43.314090 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.314054 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2" Nov 04 03:58:43.321653 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.321618 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" Nov 04 03:58:43.334851 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.333774 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-6a9cdb\"" Nov 04 03:58:43.351194 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.351158 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2"] Nov 04 03:58:43.371702 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.361074 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" event={"ID":"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1","Type":"ContainerStarted","Data":"5ff811fc04260d118de37275abef1a949f2a6b8c706404e66bd6302440726b7c"} Nov 04 03:58:43.371702 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.361862 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" Nov 04 03:58:43.371702 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.364473 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" event={"ID":"28e01b06-4f41-4ec9-b351-6145f5d0bd2a","Type":"ContainerStarted","Data":"641b44eef565e6d3d1eccc4b26631e2eef89384deddacce49e0b2c14471ab5ac"} Nov 04 03:58:43.371702 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.364532 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" event={"ID":"28e01b06-4f41-4ec9-b351-6145f5d0bd2a","Type":"ContainerStarted","Data":"7254bd2a02bee8fb4c6e3360cae55e8dbbffd2116845bd33b8119dad733309fb"} Nov 04 03:58:43.371702 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.364564 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" Nov 04 03:58:43.406633 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.406609 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-6a9cdb-dockercfg-twtnr\"" Nov 04 03:58:43.407221 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.407194 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a85861de-cb4f-4da6-ad76-4e55e957d797-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2\" (UID: \"a85861de-cb4f-4da6-ad76-4e55e957d797\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2" Nov 04 03:58:43.407544 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.407516 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj8xd\" (UniqueName: \"kubernetes.io/projected/a85861de-cb4f-4da6-ad76-4e55e957d797-kube-api-access-sj8xd\") pod \"payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2\" (UID: \"a85861de-cb4f-4da6-ad76-4e55e957d797\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2" Nov 04 03:58:43.423762 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.423181 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" podStartSLOduration=1.85546078 podStartE2EDuration="2.423160408s" totalImagesPullingTime="567.699628ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:58:43.421370408 +0000 UTC m=+567.222316217" watchObservedRunningTime="2026-11-04 03:58:43.423160408 +0000 UTC m=+567.224106217" Nov 04 03:58:43.509127 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.509061 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8xd\" (UniqueName: \"kubernetes.io/projected/a85861de-cb4f-4da6-ad76-4e55e957d797-kube-api-access-sj8xd\") pod \"payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2\" (UID: \"a85861de-cb4f-4da6-ad76-4e55e957d797\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2" Nov 04 03:58:43.509301 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.509200 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a85861de-cb4f-4da6-ad76-4e55e957d797-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2\" (UID: \"a85861de-cb4f-4da6-ad76-4e55e957d797\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2" Nov 04 03:58:43.510114 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.510085 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a85861de-cb4f-4da6-ad76-4e55e957d797-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2\" (UID: \"a85861de-cb4f-4da6-ad76-4e55e957d797\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2" Nov 04 03:58:43.530720 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.530694 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj8xd\" (UniqueName: \"kubernetes.io/projected/a85861de-cb4f-4da6-ad76-4e55e957d797-kube-api-access-sj8xd\") pod \"payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2\" (UID: \"a85861de-cb4f-4da6-ad76-4e55e957d797\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2" Nov 04 03:58:43.628994 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.628874 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2" Nov 04 03:58:43.636694 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.636651 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" podStartSLOduration=1.636635859 podStartE2EDuration="1.636635859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:58:43.464435166 +0000 UTC m=+567.265380974" watchObservedRunningTime="2026-11-04 03:58:43.636635859 +0000 UTC m=+567.437581667" Nov 04 03:58:43.636870 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.636857 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp"] Nov 04 03:58:43.640050 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:58:43.640020 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff721376_eba2_4a06_b1b9_48e5875d9c05.slice/crio-87f8c2a203c511f857bc2ad59e7ae1d663ce96d6f16b9cb3b3e650ede6ecf549 WatchSource:0}: Error finding container 87f8c2a203c511f857bc2ad59e7ae1d663ce96d6f16b9cb3b3e650ede6ecf549: Status 404 returned error can't find the container with id 87f8c2a203c511f857bc2ad59e7ae1d663ce96d6f16b9cb3b3e650ede6ecf549 Nov 04 03:58:43.923111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:43.923084 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2"] Nov 04 03:58:43.923795 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:58:43.923762 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85861de_cb4f_4da6_ad76_4e55e957d797.slice/crio-ef4ea222d36b879af0700155309672721d3e135fd9d042e323b4f1bb754a7f46 WatchSource:0}: Error finding container ef4ea222d36b879af0700155309672721d3e135fd9d042e323b4f1bb754a7f46: Status 404 returned error can't find the container with id ef4ea222d36b879af0700155309672721d3e135fd9d042e323b4f1bb754a7f46 Nov 04 03:58:44.377238 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.377202 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7"] Nov 04 03:58:44.383829 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.383726 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2" event={"ID":"a85861de-cb4f-4da6-ad76-4e55e957d797","Type":"ContainerStarted","Data":"6094f0065d70402d394fd64fa344625643a355418831a0ba651dfeb6d3703ab5"} Nov 04 03:58:44.384041 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.383810 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7" Nov 04 03:58:44.384041 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.383870 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2" Nov 04 03:58:44.384192 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.384149 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2" event={"ID":"a85861de-cb4f-4da6-ad76-4e55e957d797","Type":"ContainerStarted","Data":"ef4ea222d36b879af0700155309672721d3e135fd9d042e323b4f1bb754a7f46"} Nov 04 03:58:44.385008 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.384982 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" event={"ID":"ff721376-eba2-4a06-b1b9-48e5875d9c05","Type":"ContainerStarted","Data":"6f35ee0162dd28be19156dc2285c3ed675ee82b4ff785245527a2c7c384665a7"} Nov 04 03:58:44.385136 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.385026 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" event={"ID":"ff721376-eba2-4a06-b1b9-48e5875d9c05","Type":"ContainerStarted","Data":"87f8c2a203c511f857bc2ad59e7ae1d663ce96d6f16b9cb3b3e650ede6ecf549"} Nov 04 03:58:44.385498 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.385442 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" Nov 04 03:58:44.389877 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.389857 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-78ca37\"" Nov 04 03:58:44.390134 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.390114 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7"] Nov 04 03:58:44.391058 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.391036 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-78ca37-dockercfg-mvrtm\"" Nov 04 03:58:44.428246 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.428141 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" podStartSLOduration=1.9643087430000001 podStartE2EDuration="2.428124186s" totalImagesPullingTime="463.815443ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:58:44.427678555 +0000 UTC m=+568.228624375" watchObservedRunningTime="2026-11-04 03:58:44.428124186 +0000 UTC m=+568.229069995" Nov 04 03:58:44.430395 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.430346 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2" podStartSLOduration=1.430335279 podStartE2EDuration="1.430335279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:58:44.405891149 +0000 UTC m=+568.206836970" watchObservedRunningTime="2026-11-04 03:58:44.430335279 +0000 UTC m=+568.231281140" Nov 04 03:58:44.519643 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.519572 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5ce632-2597-47ef-bb2f-dd41f330c12f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7\" (UID: \"6d5ce632-2597-47ef-bb2f-dd41f330c12f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7" Nov 04 03:58:44.520157 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.520121 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw95q\" (UniqueName: \"kubernetes.io/projected/6d5ce632-2597-47ef-bb2f-dd41f330c12f-kube-api-access-rw95q\") pod \"payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7\" (UID: \"6d5ce632-2597-47ef-bb2f-dd41f330c12f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7" Nov 04 03:58:44.621512 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.621473 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw95q\" (UniqueName: \"kubernetes.io/projected/6d5ce632-2597-47ef-bb2f-dd41f330c12f-kube-api-access-rw95q\") pod \"payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7\" (UID: \"6d5ce632-2597-47ef-bb2f-dd41f330c12f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7" Nov 04 03:58:44.621653 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.621604 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5ce632-2597-47ef-bb2f-dd41f330c12f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7\" (UID: \"6d5ce632-2597-47ef-bb2f-dd41f330c12f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7" Nov 04 03:58:44.622346 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.622319 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5ce632-2597-47ef-bb2f-dd41f330c12f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7\" (UID: \"6d5ce632-2597-47ef-bb2f-dd41f330c12f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7" Nov 04 03:58:44.634025 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.633999 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw95q\" (UniqueName: \"kubernetes.io/projected/6d5ce632-2597-47ef-bb2f-dd41f330c12f-kube-api-access-rw95q\") pod \"payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7\" (UID: \"6d5ce632-2597-47ef-bb2f-dd41f330c12f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7" Nov 04 03:58:44.706303 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.706218 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7" Nov 04 03:58:44.781025 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.780894 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg"] Nov 04 03:58:44.787154 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.787130 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" Nov 04 03:58:44.793334 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.793302 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-36730f\"" Nov 04 03:58:44.796154 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.796082 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg"] Nov 04 03:58:44.833245 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.833213 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf"] Nov 04 03:58:44.845253 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.845220 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" Nov 04 03:58:44.851861 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.851832 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf"] Nov 04 03:58:44.863495 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.860042 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-56a44f-serving-cert\"" Nov 04 03:58:44.863495 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.860170 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-36730f-dockercfg-zmzjl\"" Nov 04 03:58:44.925361 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.925326 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsmzz\" (UniqueName: \"kubernetes.io/projected/e8517352-f4dc-4e65-8ac7-ccb0b01d5255-kube-api-access-dsmzz\") pod \"payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg\" (UID: \"e8517352-f4dc-4e65-8ac7-ccb0b01d5255\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" Nov 04 03:58:44.925519 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.925387 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8517352-f4dc-4e65-8ac7-ccb0b01d5255-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg\" (UID: \"e8517352-f4dc-4e65-8ac7-ccb0b01d5255\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" Nov 04 03:58:44.993823 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:44.993785 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7"] Nov 04 03:58:45.027093 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.027059 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9ae98de-5794-48e0-ae47-d93b38c29ab0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf\" (UID: \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" Nov 04 03:58:45.027200 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.027152 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsmzz\" (UniqueName: \"kubernetes.io/projected/e8517352-f4dc-4e65-8ac7-ccb0b01d5255-kube-api-access-dsmzz\") pod \"payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg\" (UID: \"e8517352-f4dc-4e65-8ac7-ccb0b01d5255\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" Nov 04 03:58:45.027252 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.027212 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ae98de-5794-48e0-ae47-d93b38c29ab0-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf\" (UID: \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" Nov 04 03:58:45.027284 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.027251 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8517352-f4dc-4e65-8ac7-ccb0b01d5255-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg\" (UID: \"e8517352-f4dc-4e65-8ac7-ccb0b01d5255\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" Nov 04 03:58:45.027322 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.027279 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9ae98de-5794-48e0-ae47-d93b38c29ab0-metrics-certs\") pod \"maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf\" (UID: \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" Nov 04 03:58:45.027382 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.027366 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhdcw\" (UniqueName: \"kubernetes.io/projected/b9ae98de-5794-48e0-ae47-d93b38c29ab0-kube-api-access-jhdcw\") pod \"maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf\" (UID: \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" Nov 04 03:58:45.028449 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.028424 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8517352-f4dc-4e65-8ac7-ccb0b01d5255-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg\" (UID: \"e8517352-f4dc-4e65-8ac7-ccb0b01d5255\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" Nov 04 03:58:45.039655 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.039631 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsmzz\" (UniqueName: \"kubernetes.io/projected/e8517352-f4dc-4e65-8ac7-ccb0b01d5255-kube-api-access-dsmzz\") pod \"payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg\" (UID: \"e8517352-f4dc-4e65-8ac7-ccb0b01d5255\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" Nov 04 03:58:45.118767 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.118742 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" Nov 04 03:58:45.128991 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.128965 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9ae98de-5794-48e0-ae47-d93b38c29ab0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf\" (UID: \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" Nov 04 03:58:45.129124 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.129029 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ae98de-5794-48e0-ae47-d93b38c29ab0-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf\" (UID: \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" Nov 04 03:58:45.129124 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.129074 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9ae98de-5794-48e0-ae47-d93b38c29ab0-metrics-certs\") pod \"maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf\" (UID: \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" Nov 04 03:58:45.129229 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.129153 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhdcw\" (UniqueName: \"kubernetes.io/projected/b9ae98de-5794-48e0-ae47-d93b38c29ab0-kube-api-access-jhdcw\") pod \"maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf\" (UID: \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" Nov 04 03:58:45.129804 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.129773 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ae98de-5794-48e0-ae47-d93b38c29ab0-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf\" (UID: \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" Nov 04 03:58:45.131713 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.131688 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9ae98de-5794-48e0-ae47-d93b38c29ab0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf\" (UID: \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" Nov 04 03:58:45.131796 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.131723 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9ae98de-5794-48e0-ae47-d93b38c29ab0-metrics-certs\") pod \"maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf\" (UID: \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" Nov 04 03:58:45.155846 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.155775 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhdcw\" (UniqueName: \"kubernetes.io/projected/b9ae98de-5794-48e0-ae47-d93b38c29ab0-kube-api-access-jhdcw\") pod \"maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf\" (UID: \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" Nov 04 03:58:45.178121 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.177837 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" Nov 04 03:58:45.395971 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.395932 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7" event={"ID":"6d5ce632-2597-47ef-bb2f-dd41f330c12f","Type":"ContainerStarted","Data":"ca31ca80e08b06efc1b1620cb5416a22a4c1b7b5adc4c9363b3f66d21fd065f4"} Nov 04 03:58:45.396473 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.395994 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7" event={"ID":"6d5ce632-2597-47ef-bb2f-dd41f330c12f","Type":"ContainerStarted","Data":"d6ca3d9cbfcd0dbe07e90948821167a43a76ae0901cae59308bae0fe08abe7e8"} Nov 04 03:58:45.397027 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.397007 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7" Nov 04 03:58:45.420054 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.419984 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7" podStartSLOduration=1.4199659869999999 podStartE2EDuration="1.419965987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:58:45.416710822 +0000 UTC m=+569.217656630" watchObservedRunningTime="2026-11-04 03:58:45.419965987 +0000 UTC m=+569.220911796" Nov 04 03:58:45.426143 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.426115 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg"] Nov 04 03:58:45.427408 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:58:45.427376 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8517352_f4dc_4e65_8ac7_ccb0b01d5255.slice/crio-7ce95ea356082b83444524cc4d03a0fadf88cd1803c99f3b729190c821aaa693 WatchSource:0}: Error finding container 7ce95ea356082b83444524cc4d03a0fadf88cd1803c99f3b729190c821aaa693: Status 404 returned error can't find the container with id 7ce95ea356082b83444524cc4d03a0fadf88cd1803c99f3b729190c821aaa693 Nov 04 03:58:45.493273 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:45.493245 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf"] Nov 04 03:58:45.497043 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:58:45.497003 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ae98de_5794_48e0_ae47_d93b38c29ab0.slice/crio-53bebdf68303a5107e5f55fd143b0ed86a7c9908d872f1ac5fcd1eedc8dd3a34 WatchSource:0}: Error finding container 53bebdf68303a5107e5f55fd143b0ed86a7c9908d872f1ac5fcd1eedc8dd3a34: Status 404 returned error can't find the container with id 53bebdf68303a5107e5f55fd143b0ed86a7c9908d872f1ac5fcd1eedc8dd3a34 Nov 04 03:58:46.402011 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.401956 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" event={"ID":"b9ae98de-5794-48e0-ae47-d93b38c29ab0","Type":"ContainerStarted","Data":"c097693692277ef812df5dfec02dd807044547f22510556c39546bc51d62fe4d"} Nov 04 03:58:46.402483 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.402031 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" event={"ID":"b9ae98de-5794-48e0-ae47-d93b38c29ab0","Type":"ContainerStarted","Data":"53bebdf68303a5107e5f55fd143b0ed86a7c9908d872f1ac5fcd1eedc8dd3a34"} Nov 04 03:58:46.402483 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.402062 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" Nov 04 03:58:46.403779 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.403742 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" event={"ID":"e8517352-f4dc-4e65-8ac7-ccb0b01d5255","Type":"ContainerStarted","Data":"398b12ee0e5eceb08f69a5e4482c133a7a718d9dce147060980cf364bf6b2228"} Nov 04 03:58:46.403943 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.403795 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" event={"ID":"e8517352-f4dc-4e65-8ac7-ccb0b01d5255","Type":"ContainerStarted","Data":"7ce95ea356082b83444524cc4d03a0fadf88cd1803c99f3b729190c821aaa693"} Nov 04 03:58:46.404186 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.404168 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" Nov 04 03:58:46.440513 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.440460 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" podStartSLOduration=2.013680182 podStartE2EDuration="2.440446132s" totalImagesPullingTime="426.76595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:58:46.439388714 +0000 UTC m=+570.240334744" watchObservedRunningTime="2026-11-04 03:58:46.440446132 +0000 UTC m=+570.241391954" Nov 04 03:58:46.475603 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.475546 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" podStartSLOduration=2.475533385 podStartE2EDuration="2.475533385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:58:46.474349893 +0000 UTC m=+570.275295701" watchObservedRunningTime="2026-11-04 03:58:46.475533385 +0000 UTC m=+570.276479194" Nov 04 03:58:46.831309 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.831270 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms"] Nov 04 03:58:46.836413 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.836384 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" Nov 04 03:58:46.852913 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.850479 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms"] Nov 04 03:58:46.852913 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.850832 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-78ca37-serving-cert\"" Nov 04 03:58:46.852913 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.851882 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016f8a5d-06ed-4e0c-9a36-f541c1232337-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms\" (UID: \"016f8a5d-06ed-4e0c-9a36-f541c1232337\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" Nov 04 03:58:46.852913 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.851939 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdbgs\" (UniqueName: \"kubernetes.io/projected/016f8a5d-06ed-4e0c-9a36-f541c1232337-kube-api-access-gdbgs\") pod \"maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms\" (UID: \"016f8a5d-06ed-4e0c-9a36-f541c1232337\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" Nov 04 03:58:46.852913 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.852067 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016f8a5d-06ed-4e0c-9a36-f541c1232337-maas-api-tls\") pod \"maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms\" (UID: \"016f8a5d-06ed-4e0c-9a36-f541c1232337\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" Nov 04 03:58:46.852913 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.852150 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016f8a5d-06ed-4e0c-9a36-f541c1232337-metrics-certs\") pod \"maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms\" (UID: \"016f8a5d-06ed-4e0c-9a36-f541c1232337\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" Nov 04 03:58:46.953112 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.953070 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016f8a5d-06ed-4e0c-9a36-f541c1232337-maas-api-tls\") pod \"maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms\" (UID: \"016f8a5d-06ed-4e0c-9a36-f541c1232337\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" Nov 04 03:58:46.953321 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.953156 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016f8a5d-06ed-4e0c-9a36-f541c1232337-metrics-certs\") pod \"maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms\" (UID: \"016f8a5d-06ed-4e0c-9a36-f541c1232337\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" Nov 04 03:58:46.953391 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.953316 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016f8a5d-06ed-4e0c-9a36-f541c1232337-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms\" (UID: \"016f8a5d-06ed-4e0c-9a36-f541c1232337\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" Nov 04 03:58:46.953391 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.953348 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdbgs\" (UniqueName: \"kubernetes.io/projected/016f8a5d-06ed-4e0c-9a36-f541c1232337-kube-api-access-gdbgs\") pod \"maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms\" (UID: \"016f8a5d-06ed-4e0c-9a36-f541c1232337\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" Nov 04 03:58:46.953391 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:58:46.953383 4966 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-serving-cert: secret "maas-api-e2e-worker-w2-78ca37-serving-cert" not found Nov 04 03:58:46.953540 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:58:46.953448 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/016f8a5d-06ed-4e0c-9a36-f541c1232337-maas-api-tls podName:016f8a5d-06ed-4e0c-9a36-f541c1232337 nodeName:}" failed. No retries permitted until 2026-11-04 03:58:47.453429473 +0000 UTC m=+571.254375272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/016f8a5d-06ed-4e0c-9a36-f541c1232337-maas-api-tls") pod "maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" (UID: "016f8a5d-06ed-4e0c-9a36-f541c1232337") : secret "maas-api-e2e-worker-w2-78ca37-serving-cert" not found Nov 04 03:58:46.954262 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.954238 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016f8a5d-06ed-4e0c-9a36-f541c1232337-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms\" (UID: \"016f8a5d-06ed-4e0c-9a36-f541c1232337\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" Nov 04 03:58:46.956045 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.956023 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016f8a5d-06ed-4e0c-9a36-f541c1232337-metrics-certs\") pod \"maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms\" (UID: \"016f8a5d-06ed-4e0c-9a36-f541c1232337\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" Nov 04 03:58:46.968500 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:46.968473 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdbgs\" (UniqueName: \"kubernetes.io/projected/016f8a5d-06ed-4e0c-9a36-f541c1232337-kube-api-access-gdbgs\") pod \"maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms\" (UID: \"016f8a5d-06ed-4e0c-9a36-f541c1232337\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" Nov 04 03:58:47.459401 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.459369 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016f8a5d-06ed-4e0c-9a36-f541c1232337-maas-api-tls\") pod \"maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms\" (UID: \"016f8a5d-06ed-4e0c-9a36-f541c1232337\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" Nov 04 03:58:47.461758 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.461731 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016f8a5d-06ed-4e0c-9a36-f541c1232337-maas-api-tls\") pod \"maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms\" (UID: \"016f8a5d-06ed-4e0c-9a36-f541c1232337\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" Nov 04 03:58:47.462562 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.462542 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" Nov 04 03:58:47.754356 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.754109 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms"] Nov 04 03:58:47.757420 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:58:47.757392 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016f8a5d_06ed_4e0c_9a36_f541c1232337.slice/crio-1076cfe62518d06ba08847c17299d287777b2d9db26a657089ab583ce00d0ed7 WatchSource:0}: Error finding container 1076cfe62518d06ba08847c17299d287777b2d9db26a657089ab583ce00d0ed7: Status 404 returned error can't find the container with id 1076cfe62518d06ba08847c17299d287777b2d9db26a657089ab583ce00d0ed7 Nov 04 03:58:47.838852 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.838807 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v"] Nov 04 03:58:47.844224 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.844201 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" Nov 04 03:58:47.854881 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.854854 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-36730f-serving-cert\"" Nov 04 03:58:47.861021 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.860999 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v"] Nov 04 03:58:47.863169 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.863144 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-metrics-certs\") pod \"maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v\" (UID: \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" Nov 04 03:58:47.863263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.863204 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-maas-api-tls\") pod \"maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v\" (UID: \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" Nov 04 03:58:47.863450 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.863425 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbss2\" (UniqueName: \"kubernetes.io/projected/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-kube-api-access-kbss2\") pod \"maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v\" (UID: \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" Nov 04 03:58:47.863511 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.863494 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v\" (UID: \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" Nov 04 03:58:47.964506 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.964476 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbss2\" (UniqueName: \"kubernetes.io/projected/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-kube-api-access-kbss2\") pod \"maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v\" (UID: \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" Nov 04 03:58:47.964681 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.964532 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v\" (UID: \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" Nov 04 03:58:47.964681 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.964591 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-metrics-certs\") pod \"maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v\" (UID: \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" Nov 04 03:58:47.964681 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.964636 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-maas-api-tls\") pod \"maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v\" (UID: \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" Nov 04 03:58:47.965470 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.965402 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v\" (UID: \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" Nov 04 03:58:47.967209 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.967189 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-metrics-certs\") pod \"maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v\" (UID: \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" Nov 04 03:58:47.967320 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.967253 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-maas-api-tls\") pod \"maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v\" (UID: \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" Nov 04 03:58:47.976770 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:47.976736 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbss2\" (UniqueName: \"kubernetes.io/projected/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-kube-api-access-kbss2\") pod \"maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v\" (UID: \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" Nov 04 03:58:48.161625 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:48.161589 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" Nov 04 03:58:48.420326 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:48.419485 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" event={"ID":"016f8a5d-06ed-4e0c-9a36-f541c1232337","Type":"ContainerStarted","Data":"afdc154ef92311df143b3988b77416cde575ea3cb12aea4d93a07c536829bac8"} Nov 04 03:58:48.420326 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:48.419559 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" event={"ID":"016f8a5d-06ed-4e0c-9a36-f541c1232337","Type":"ContainerStarted","Data":"1076cfe62518d06ba08847c17299d287777b2d9db26a657089ab583ce00d0ed7"} Nov 04 03:58:48.420326 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:48.419969 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" Nov 04 03:58:48.444504 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:48.444440 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" podStartSLOduration=1.908412969 podStartE2EDuration="2.444421292s" totalImagesPullingTime="536.008323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:58:48.442091195 +0000 UTC m=+572.243037004" watchObservedRunningTime="2026-11-04 03:58:48.444421292 +0000 UTC m=+572.245367100" Nov 04 03:58:48.450294 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:48.450260 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v"] Nov 04 03:58:48.450809 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:58:48.450780 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd910f8fa_7ab2_44c5_b175_a56bf0e0d6b2.slice/crio-40ccd6b1270da18b3ea9523ce0a75806b48dc316652a02005d8d304df0e08bce WatchSource:0}: Error finding container 40ccd6b1270da18b3ea9523ce0a75806b48dc316652a02005d8d304df0e08bce: Status 404 returned error can't find the container with id 40ccd6b1270da18b3ea9523ce0a75806b48dc316652a02005d8d304df0e08bce Nov 04 03:58:49.434732 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:49.434694 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" event={"ID":"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2","Type":"ContainerStarted","Data":"8689d8433b360a49de25d939a46622d55731c4c8bae8d8a4d765293fd96edd3c"} Nov 04 03:58:49.435154 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:49.434769 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" event={"ID":"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2","Type":"ContainerStarted","Data":"40ccd6b1270da18b3ea9523ce0a75806b48dc316652a02005d8d304df0e08bce"} Nov 04 03:58:49.459425 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:49.459361 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" podStartSLOduration=2.067628771 podStartE2EDuration="2.459342574s" totalImagesPullingTime="391.713803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:58:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:58:49.455738466 +0000 UTC m=+573.256684275" watchObservedRunningTime="2026-11-04 03:58:49.459342574 +0000 UTC m=+573.260288382" Nov 04 03:58:50.402892 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:50.402864 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" Nov 04 03:58:50.442566 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:50.442535 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" Nov 04 03:58:51.410296 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:51.410260 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" Nov 04 03:58:52.347822 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:52.347791 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs" Nov 04 03:58:52.348272 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:52.347868 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn" Nov 04 03:58:52.415552 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:52.415523 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" Nov 04 03:58:53.368305 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:53.368267 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6" Nov 04 03:58:53.368803 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:53.368336 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" Nov 04 03:58:54.388571 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:54.388542 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" Nov 04 03:58:54.441060 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:54.441030 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" Nov 04 03:58:56.407551 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:56.407524 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2" Nov 04 03:58:56.458511 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:56.458483 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" Nov 04 03:58:57.413487 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:57.413454 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7" Nov 04 03:58:58.424043 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:58:58.423811 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" Nov 04 03:59:05.677530 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:59:05.677488 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:59:16.428174 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:16.428140 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/3.log" Nov 04 03:59:16.431020 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:16.430993 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/3.log" Nov 04 03:59:32.700493 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.700459 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw"] Nov 04 03:59:32.704516 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.704495 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:32.712146 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.712048 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-998997\"/\"kube-root-ca.crt\"" Nov 04 03:59:32.712146 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.712113 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-998997\"/\"openshift-service-ca.crt\"" Nov 04 03:59:32.712333 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.712292 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-998997\"/\"default-dockercfg-jqfqv\"" Nov 04 03:59:32.712423 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.712396 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-998997\"/\"premium-w3-998997-kserve-self-signed-certs\"" Nov 04 03:59:32.715197 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.715173 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw"] Nov 04 03:59:32.781572 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.781539 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-789qg\" (UniqueName: \"kubernetes.io/projected/6646d4de-0e96-4e6d-91f9-a426a7a862b4-kube-api-access-789qg\") pod \"premium-w3-998997-kserve-5d64f845fb-pt4gw\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:32.781719 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.781614 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-home\") pod \"premium-w3-998997-kserve-5d64f845fb-pt4gw\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:32.781719 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.781690 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-model-cache\") pod \"premium-w3-998997-kserve-5d64f845fb-pt4gw\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:32.781796 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.781722 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-dshm\") pod \"premium-w3-998997-kserve-5d64f845fb-pt4gw\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:32.781796 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.781742 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6646d4de-0e96-4e6d-91f9-a426a7a862b4-tls-certs\") pod \"premium-w3-998997-kserve-5d64f845fb-pt4gw\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:32.781796 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.781762 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-tmp-dir\") pod \"premium-w3-998997-kserve-5d64f845fb-pt4gw\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:32.882678 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.882638 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-789qg\" (UniqueName: \"kubernetes.io/projected/6646d4de-0e96-4e6d-91f9-a426a7a862b4-kube-api-access-789qg\") pod \"premium-w3-998997-kserve-5d64f845fb-pt4gw\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:32.882853 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.882705 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-home\") pod \"premium-w3-998997-kserve-5d64f845fb-pt4gw\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:32.882853 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.882756 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-model-cache\") pod \"premium-w3-998997-kserve-5d64f845fb-pt4gw\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:32.882853 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.882784 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-dshm\") pod \"premium-w3-998997-kserve-5d64f845fb-pt4gw\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:32.882853 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.882801 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6646d4de-0e96-4e6d-91f9-a426a7a862b4-tls-certs\") pod \"premium-w3-998997-kserve-5d64f845fb-pt4gw\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:32.882853 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.882822 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-tmp-dir\") pod \"premium-w3-998997-kserve-5d64f845fb-pt4gw\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:32.883174 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.883152 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-home\") pod \"premium-w3-998997-kserve-5d64f845fb-pt4gw\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:32.883242 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.883204 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-tmp-dir\") pod \"premium-w3-998997-kserve-5d64f845fb-pt4gw\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:32.883242 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.883223 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-model-cache\") pod \"premium-w3-998997-kserve-5d64f845fb-pt4gw\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:32.885366 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.885343 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6646d4de-0e96-4e6d-91f9-a426a7a862b4-tls-certs\") pod \"premium-w3-998997-kserve-5d64f845fb-pt4gw\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:32.885474 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.885387 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-dshm\") pod \"premium-w3-998997-kserve-5d64f845fb-pt4gw\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:32.896584 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:32.896559 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-789qg\" (UniqueName: \"kubernetes.io/projected/6646d4de-0e96-4e6d-91f9-a426a7a862b4-kube-api-access-789qg\") pod \"premium-w3-998997-kserve-5d64f845fb-pt4gw\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:33.019241 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:33.019165 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:33.301793 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:33.301764 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw"] Nov 04 03:59:33.302617 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:59:33.302577 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6646d4de_0e96_4e6d_91f9_a426a7a862b4.slice/crio-2c612fd7a82b989785339debf7ba7b981fb3616a061472af088a01791ba7608f WatchSource:0}: Error finding container 2c612fd7a82b989785339debf7ba7b981fb3616a061472af088a01791ba7608f: Status 404 returned error can't find the container with id 2c612fd7a82b989785339debf7ba7b981fb3616a061472af088a01791ba7608f Nov 04 03:59:33.746725 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:33.746689 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" event={"ID":"6646d4de-0e96-4e6d-91f9-a426a7a862b4","Type":"ContainerStarted","Data":"fd574be7a86fb72def83728d94322c91f226444712c0b48c586c45646b6bbac7"} Nov 04 03:59:33.747111 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:33.746744 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" event={"ID":"6646d4de-0e96-4e6d-91f9-a426a7a862b4","Type":"ContainerStarted","Data":"2c612fd7a82b989785339debf7ba7b981fb3616a061472af088a01791ba7608f"} Nov 04 03:59:33.769056 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:33.769003 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" podStartSLOduration=1.768989683 podStartE2EDuration="1.768989683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:59:33.766211097 +0000 UTC m=+617.567156907" watchObservedRunningTime="2026-11-04 03:59:33.768989683 +0000 UTC m=+617.569935491" Nov 04 03:59:34.020143 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:34.020061 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:35.677285 ip-10-0-138-9 kubenswrapper[4966]: E1104 03:59:35.677246 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 03:59:43.020164 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:43.020129 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:43.033058 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:43.033028 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:43.821937 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:43.821898 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 03:59:52.212442 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.212408 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh"] Nov 04 03:59:52.217878 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.217859 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.225326 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.225299 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-64c20d\"/\"openshift-service-ca.crt\"" Nov 04 03:59:52.225579 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.225331 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-64c20d\"/\"kube-root-ca.crt\"" Nov 04 03:59:52.227161 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.227108 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-64c20d\"/\"premium-w4-64c20d-kserve-self-signed-certs\"" Nov 04 03:59:52.227275 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.227194 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-64c20d\"/\"default-dockercfg-5mf6f\"" Nov 04 03:59:52.231147 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.231120 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh"] Nov 04 03:59:52.306993 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.306950 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4bxh\" (UniqueName: \"kubernetes.io/projected/f939b53c-8938-4bb4-b334-7e329d1c366d-kube-api-access-m4bxh\") pod \"premium-w4-64c20d-kserve-9598d8bf5-cw5rh\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.307181 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.307077 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-home\") pod \"premium-w4-64c20d-kserve-9598d8bf5-cw5rh\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.307181 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.307110 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f939b53c-8938-4bb4-b334-7e329d1c366d-tls-certs\") pod \"premium-w4-64c20d-kserve-9598d8bf5-cw5rh\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.307263 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.307216 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-model-cache\") pod \"premium-w4-64c20d-kserve-9598d8bf5-cw5rh\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.307298 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.307278 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-dshm\") pod \"premium-w4-64c20d-kserve-9598d8bf5-cw5rh\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.307345 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.307314 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-tmp-dir\") pod \"premium-w4-64c20d-kserve-9598d8bf5-cw5rh\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.408972 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.408902 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4bxh\" (UniqueName: \"kubernetes.io/projected/f939b53c-8938-4bb4-b334-7e329d1c366d-kube-api-access-m4bxh\") pod \"premium-w4-64c20d-kserve-9598d8bf5-cw5rh\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.409170 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.409006 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-home\") pod \"premium-w4-64c20d-kserve-9598d8bf5-cw5rh\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.409170 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.409029 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f939b53c-8938-4bb4-b334-7e329d1c366d-tls-certs\") pod \"premium-w4-64c20d-kserve-9598d8bf5-cw5rh\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.409170 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.409050 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-model-cache\") pod \"premium-w4-64c20d-kserve-9598d8bf5-cw5rh\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.409170 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.409129 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-dshm\") pod \"premium-w4-64c20d-kserve-9598d8bf5-cw5rh\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.409411 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.409187 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-tmp-dir\") pod \"premium-w4-64c20d-kserve-9598d8bf5-cw5rh\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.409511 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.409486 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-home\") pod \"premium-w4-64c20d-kserve-9598d8bf5-cw5rh\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.409570 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.409502 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-model-cache\") pod \"premium-w4-64c20d-kserve-9598d8bf5-cw5rh\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.409662 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.409643 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-tmp-dir\") pod \"premium-w4-64c20d-kserve-9598d8bf5-cw5rh\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.411504 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.411479 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-dshm\") pod \"premium-w4-64c20d-kserve-9598d8bf5-cw5rh\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.411685 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.411668 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f939b53c-8938-4bb4-b334-7e329d1c366d-tls-certs\") pod \"premium-w4-64c20d-kserve-9598d8bf5-cw5rh\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.421282 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.421260 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4bxh\" (UniqueName: \"kubernetes.io/projected/f939b53c-8938-4bb4-b334-7e329d1c366d-kube-api-access-m4bxh\") pod \"premium-w4-64c20d-kserve-9598d8bf5-cw5rh\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.531782 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.531696 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 03:59:52.813044 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.813016 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh"] Nov 04 03:59:52.815185 ip-10-0-138-9 kubenswrapper[4966]: W1104 03:59:52.815158 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf939b53c_8938_4bb4_b334_7e329d1c366d.slice/crio-272d475a7cd384b96eb29e69ce5d92983ca782b797fdd3602130443ac070ef99 WatchSource:0}: Error finding container 272d475a7cd384b96eb29e69ce5d92983ca782b797fdd3602130443ac070ef99: Status 404 returned error can't find the container with id 272d475a7cd384b96eb29e69ce5d92983ca782b797fdd3602130443ac070ef99 Nov 04 03:59:52.874163 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:52.874131 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" event={"ID":"f939b53c-8938-4bb4-b334-7e329d1c366d","Type":"ContainerStarted","Data":"272d475a7cd384b96eb29e69ce5d92983ca782b797fdd3602130443ac070ef99"} Nov 04 03:59:53.882672 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:53.882637 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" event={"ID":"f939b53c-8938-4bb4-b334-7e329d1c366d","Type":"ContainerStarted","Data":"11f4b7ea62ded68b45eaed7ec2a9b4c165492c148879a70cfc494362c7fb7095"} Nov 04 03:59:53.903579 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:53.903529 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" podStartSLOduration=1.903515423 podStartE2EDuration="1.903515423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:59:53.901325355 +0000 UTC m=+637.702271163" watchObservedRunningTime="2026-11-04 03:59:53.903515423 +0000 UTC m=+637.704461230" Nov 04 03:59:54.532107 ip-10-0-138-9 kubenswrapper[4966]: I1104 03:59:54.532069 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 04:00:02.532146 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:02.532103 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 04:00:02.545158 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:02.545126 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 04:00:02.948968 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:02.948910 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 04:00:05.680008 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:00:05.679970 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 04:00:35.470091 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.470006 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf"] Nov 04 04:00:35.474209 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.474182 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.482668 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.482641 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-64c20d-openshift-default-dockercfg-xpqv6\"" Nov 04 04:00:35.492885 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.492855 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf"] Nov 04 04:00:35.535510 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.535470 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-workload-certs\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.535659 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.535531 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-credential-socket\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.535659 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.535601 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-podinfo\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.535659 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.535655 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-workload-socket\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.535815 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.535677 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shzjn\" (UniqueName: \"kubernetes.io/projected/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-kube-api-access-shzjn\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.535872 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.535838 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-envoy\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.535951 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.535871 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-data\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.536001 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.535946 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-token\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.536042 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.536007 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istiod-ca-cert\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.637061 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.637024 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istiod-ca-cert\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.637250 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.637094 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-workload-certs\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.637250 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.637155 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-credential-socket\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.637250 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.637179 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-podinfo\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.637250 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.637219 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-workload-socket\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.637472 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.637267 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shzjn\" (UniqueName: \"kubernetes.io/projected/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-kube-api-access-shzjn\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.637472 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.637348 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-envoy\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.637472 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.637375 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-data\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.637472 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.637403 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-token\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.637675 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.637547 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-credential-socket\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.637675 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.637607 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-workload-certs\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.637675 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.637634 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-workload-socket\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.637855 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.637833 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istiod-ca-cert\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.637993 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.637878 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-data\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.639575 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.639553 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-envoy\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.639716 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.639696 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-podinfo\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.649501 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.649474 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-token\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.649732 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.649714 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shzjn\" (UniqueName: \"kubernetes.io/projected/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-kube-api-access-shzjn\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:35.678036 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:00:35.677997 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 04:00:35.796479 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:35.796385 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:36.092939 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:36.092073 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf"] Nov 04 04:00:36.101126 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:36.098236 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:00:36.101126 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:36.098321 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:00:36.101126 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:36.098362 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:00:36.160228 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:36.160201 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" event={"ID":"8ed1dacd-75d6-4e61-94bb-d7fcced94f85","Type":"ContainerStarted","Data":"9ca910132520cbe85883377df3449b7affeee6132c2f1b59787baa87988f159b"} Nov 04 04:00:37.183966 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:37.183893 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" event={"ID":"8ed1dacd-75d6-4e61-94bb-d7fcced94f85","Type":"ContainerStarted","Data":"274b102a5dfea0df566582f92ebdc426fa29428b46ded6491344cd3251e0e5ce"} Nov 04 04:00:37.210142 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:37.210082 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" podStartSLOduration=2.210062778 podStartE2EDuration="2.210062778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:00:37.207787348 +0000 UTC m=+681.008733156" watchObservedRunningTime="2026-11-04 04:00:37.210062778 +0000 UTC m=+681.011008587" Nov 04 04:00:37.796584 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:37.796549 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:37.798305 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:37.798279 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 04:00:37.798417 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:37.798335 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" podUID="8ed1dacd-75d6-4e61-94bb-d7fcced94f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 04:00:38.797196 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:38.797162 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 04:00:38.797595 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:38.797228 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" podUID="8ed1dacd-75d6-4e61-94bb-d7fcced94f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 04:00:39.589883 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.589825 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw"] Nov 04 04:00:39.599901 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.599873 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:39.615678 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.615392 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-56a44f\"/\"default-dockercfg-vfh6g\"" Nov 04 04:00:39.615678 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.615501 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-56a44f\"/\"distinct-w5-56a44f-kserve-self-signed-certs\"" Nov 04 04:00:39.615678 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.615572 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-56a44f\"/\"kube-root-ca.crt\"" Nov 04 04:00:39.615678 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.615641 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-56a44f\"/\"openshift-service-ca.crt\"" Nov 04 04:00:39.617060 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.617036 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw"] Nov 04 04:00:39.706111 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.706065 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed1e3d9-e514-4d8f-b1d3-851453900699-tls-certs\") pod \"distinct-w5-56a44f-kserve-6d69786df6-4n9dw\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:39.706275 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.706134 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-model-cache\") pod \"distinct-w5-56a44f-kserve-6d69786df6-4n9dw\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:39.706275 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.706193 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-dshm\") pod \"distinct-w5-56a44f-kserve-6d69786df6-4n9dw\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:39.706275 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.706225 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-home\") pod \"distinct-w5-56a44f-kserve-6d69786df6-4n9dw\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:39.706275 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.706244 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxh9b\" (UniqueName: \"kubernetes.io/projected/eed1e3d9-e514-4d8f-b1d3-851453900699-kube-api-access-pxh9b\") pod \"distinct-w5-56a44f-kserve-6d69786df6-4n9dw\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:39.706413 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.706295 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-tmp-dir\") pod \"distinct-w5-56a44f-kserve-6d69786df6-4n9dw\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:39.803944 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.800345 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 04:00:39.803944 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.800418 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" podUID="8ed1dacd-75d6-4e61-94bb-d7fcced94f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 04:00:39.811049 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.810185 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-tmp-dir\") pod \"distinct-w5-56a44f-kserve-6d69786df6-4n9dw\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:39.811049 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.810250 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed1e3d9-e514-4d8f-b1d3-851453900699-tls-certs\") pod \"distinct-w5-56a44f-kserve-6d69786df6-4n9dw\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:39.811049 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.810323 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-model-cache\") pod \"distinct-w5-56a44f-kserve-6d69786df6-4n9dw\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:39.811049 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.810419 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-dshm\") pod \"distinct-w5-56a44f-kserve-6d69786df6-4n9dw\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:39.811049 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.810470 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-home\") pod \"distinct-w5-56a44f-kserve-6d69786df6-4n9dw\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:39.811049 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.810497 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxh9b\" (UniqueName: \"kubernetes.io/projected/eed1e3d9-e514-4d8f-b1d3-851453900699-kube-api-access-pxh9b\") pod \"distinct-w5-56a44f-kserve-6d69786df6-4n9dw\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:39.811572 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.811289 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-tmp-dir\") pod \"distinct-w5-56a44f-kserve-6d69786df6-4n9dw\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:39.812276 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.812248 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-home\") pod \"distinct-w5-56a44f-kserve-6d69786df6-4n9dw\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:39.812736 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.812715 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-model-cache\") pod \"distinct-w5-56a44f-kserve-6d69786df6-4n9dw\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:39.814977 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.814956 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed1e3d9-e514-4d8f-b1d3-851453900699-tls-certs\") pod \"distinct-w5-56a44f-kserve-6d69786df6-4n9dw\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:39.819038 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.819016 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-dshm\") pod \"distinct-w5-56a44f-kserve-6d69786df6-4n9dw\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:39.825954 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.825908 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxh9b\" (UniqueName: \"kubernetes.io/projected/eed1e3d9-e514-4d8f-b1d3-851453900699-kube-api-access-pxh9b\") pod \"distinct-w5-56a44f-kserve-6d69786df6-4n9dw\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:39.937277 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:39.937134 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:40.294285 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:40.294256 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw"] Nov 04 04:00:40.295097 ip-10-0-138-9 kubenswrapper[4966]: W1104 04:00:40.295066 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeed1e3d9_e514_4d8f_b1d3_851453900699.slice/crio-b546b480d6d1b4fa7424f5044833c509d7f47e127c2cc3d904804c262b5e4b62 WatchSource:0}: Error finding container b546b480d6d1b4fa7424f5044833c509d7f47e127c2cc3d904804c262b5e4b62: Status 404 returned error can't find the container with id b546b480d6d1b4fa7424f5044833c509d7f47e127c2cc3d904804c262b5e4b62 Nov 04 04:00:40.798655 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:40.797554 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 04:00:40.798655 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:40.797611 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" podUID="8ed1dacd-75d6-4e61-94bb-d7fcced94f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 04:00:41.213854 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:41.213812 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" event={"ID":"eed1e3d9-e514-4d8f-b1d3-851453900699","Type":"ContainerStarted","Data":"ae9c302fde91d5bede7b0be5e4d5042de4338188dbabff451a02fd53f4e578f2"} Nov 04 04:00:41.213854 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:41.213861 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" event={"ID":"eed1e3d9-e514-4d8f-b1d3-851453900699","Type":"ContainerStarted","Data":"b546b480d6d1b4fa7424f5044833c509d7f47e127c2cc3d904804c262b5e4b62"} Nov 04 04:00:41.235353 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:41.235301 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" podStartSLOduration=2.235284801 podStartE2EDuration="2.235284801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:00:41.232862377 +0000 UTC m=+685.033808186" watchObservedRunningTime="2026-11-04 04:00:41.235284801 +0000 UTC m=+685.036230609" Nov 04 04:00:41.797243 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:41.797210 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 04:00:41.797431 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:41.797262 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" podUID="8ed1dacd-75d6-4e61-94bb-d7fcced94f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 04:00:41.938461 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:41.938424 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:42.796889 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:42.796849 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 04:00:42.797318 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:42.796944 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" podUID="8ed1dacd-75d6-4e61-94bb-d7fcced94f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 04:00:42.825364 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:42.825323 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6863-658bdb"] Nov 04 04:00:42.829563 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:42.829536 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6863-658bdb" Nov 04 04:00:42.834302 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:42.834272 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zg68j\"" Nov 04 04:00:42.843554 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:42.843530 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6863-658bdb"] Nov 04 04:00:42.949619 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:42.949587 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpt9k\" (UniqueName: \"kubernetes.io/projected/5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f-kube-api-access-qpt9k\") pod \"test-curl-6863-658bdb\" (UID: \"5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f\") " pod="opendatahub/test-curl-6863-658bdb" Nov 04 04:00:43.051533 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:43.051431 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpt9k\" (UniqueName: \"kubernetes.io/projected/5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f-kube-api-access-qpt9k\") pod \"test-curl-6863-658bdb\" (UID: \"5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f\") " pod="opendatahub/test-curl-6863-658bdb" Nov 04 04:00:43.065572 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:43.065534 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpt9k\" (UniqueName: \"kubernetes.io/projected/5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f-kube-api-access-qpt9k\") pod \"test-curl-6863-658bdb\" (UID: \"5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f\") " pod="opendatahub/test-curl-6863-658bdb" Nov 04 04:00:43.149892 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:43.149853 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6863-658bdb" Nov 04 04:00:43.428840 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:43.428805 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6863-658bdb"] Nov 04 04:00:43.430162 ip-10-0-138-9 kubenswrapper[4966]: W1104 04:00:43.430136 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b12f2c6_e1e2_4b2d_9fcf_84099997ee0f.slice/crio-5bb6696d84db15d1b1dabfb7bbe866e9471d1fcf138dd3718987fac1b83a5473 WatchSource:0}: Error finding container 5bb6696d84db15d1b1dabfb7bbe866e9471d1fcf138dd3718987fac1b83a5473: Status 404 returned error can't find the container with id 5bb6696d84db15d1b1dabfb7bbe866e9471d1fcf138dd3718987fac1b83a5473 Nov 04 04:00:43.433091 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:43.433069 4966 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 04:00:43.801444 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:43.801365 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:43.801883 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:43.801672 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:43.802511 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:43.802491 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:00:44.240318 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:44.240280 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6863-658bdb" event={"ID":"5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f","Type":"ContainerStarted","Data":"5bb6696d84db15d1b1dabfb7bbe866e9471d1fcf138dd3718987fac1b83a5473"} Nov 04 04:00:45.248009 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:45.247972 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6863-658bdb" event={"ID":"5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f","Type":"ContainerStarted","Data":"209cb4ac2872244cb1fdd59b1450de9eeb69e4f54b622db8914603982af40fca"} Nov 04 04:00:45.266860 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:45.266806 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6863-658bdb" podStartSLOduration=2.199954778 podStartE2EDuration="3.266794005s" totalImagesPullingTime="1.066839227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:00:45.264976887 +0000 UTC m=+689.065922689" watchObservedRunningTime="2026-11-04 04:00:45.266794005 +0000 UTC m=+689.067739812" Nov 04 04:00:45.914732 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:45.914697 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6863-658bdb"] Nov 04 04:00:45.919539 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:45.919507 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6863-658bdb"] Nov 04 04:00:47.260529 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:47.260463 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6863-658bdb" podUID="5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f" containerName="test-curl-6863-658bdb" containerID="cri-o://209cb4ac2872244cb1fdd59b1450de9eeb69e4f54b622db8914603982af40fca" gracePeriod=2 Nov 04 04:00:49.461584 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:49.461554 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6863-658bdb" Nov 04 04:00:49.526396 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:49.526323 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f-kube-api-access-qpt9k\" (UniqueName: \"kubernetes.io/projected/5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f-kube-api-access-qpt9k\") pod \"5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f\" (UID: \"5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f\") " Nov 04 04:00:49.528320 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:49.528296 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f-kube-api-access-qpt9k" pod "5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f" (UID: "5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f"). InnerVolumeSpecName "kube-api-access-qpt9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:00:49.628064 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:49.628025 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpt9k\" (UniqueName: \"kubernetes.io/projected/5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f-kube-api-access-qpt9k\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:00:49.937438 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:49.937407 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:49.950607 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:49.950580 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:50.284098 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:50.284019 4966 generic.go:356] "Generic (PLEG): container finished" podID="5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f" containerID="209cb4ac2872244cb1fdd59b1450de9eeb69e4f54b622db8914603982af40fca" exitCode=137 Nov 04 04:00:50.284098 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:50.284073 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6863-658bdb" Nov 04 04:00:50.284319 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:50.284112 4966 scope.go:122] "RemoveContainer" containerID="209cb4ac2872244cb1fdd59b1450de9eeb69e4f54b622db8914603982af40fca" Nov 04 04:00:50.294930 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:50.294892 4966 scope.go:122] "RemoveContainer" containerID="209cb4ac2872244cb1fdd59b1450de9eeb69e4f54b622db8914603982af40fca" Nov 04 04:00:50.295259 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:00:50.295236 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"209cb4ac2872244cb1fdd59b1450de9eeb69e4f54b622db8914603982af40fca\": container with ID starting with 209cb4ac2872244cb1fdd59b1450de9eeb69e4f54b622db8914603982af40fca not found: ID does not exist" containerID="209cb4ac2872244cb1fdd59b1450de9eeb69e4f54b622db8914603982af40fca" Nov 04 04:00:50.295344 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:50.295273 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209cb4ac2872244cb1fdd59b1450de9eeb69e4f54b622db8914603982af40fca"} err="failed to get container status \"209cb4ac2872244cb1fdd59b1450de9eeb69e4f54b622db8914603982af40fca\": rpc error: code = NotFound desc = could not find container \"209cb4ac2872244cb1fdd59b1450de9eeb69e4f54b622db8914603982af40fca\": container with ID starting with 209cb4ac2872244cb1fdd59b1450de9eeb69e4f54b622db8914603982af40fca not found: ID does not exist" Nov 04 04:00:50.296677 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:50.296641 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:00:50.296991 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:00:50.296958 4966 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6863-658bdb\" is forbidden: User \"system:node:ip-10-0-138-9.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-9.us-east-2.compute.internal' and this object" podUID="5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f" pod="opendatahub/test-curl-6863-658bdb" Nov 04 04:00:50.299658 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:00:50.299636 4966 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6863-658bdb\" is forbidden: User \"system:node:ip-10-0-138-9.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-9.us-east-2.compute.internal' and this object" podUID="5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f" pod="opendatahub/test-curl-6863-658bdb" Nov 04 04:00:50.344057 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:50.344022 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f" path="/var/lib/kubelet/pods/5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f/volumes" Nov 04 04:00:51.086471 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.086433 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl"] Nov 04 04:00:51.087117 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.087099 4966 cpu_manager.go:414] "containerMap: removing container" podUID="5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f" containerName="test-curl-6863-658bdb" Nov 04 04:00:51.087171 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.087118 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f" containerName="test-curl-6863-658bdb" Nov 04 04:00:51.087211 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.087205 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b12f2c6-e1e2-4b2d-9fcf-84099997ee0f" containerName="test-curl-6863-658bdb" Nov 04 04:00:51.092058 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.092034 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.106274 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.106245 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl"] Nov 04 04:00:51.183336 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.183301 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-6a9cdb-openshift-default-dockercfg-t8tng\"" Nov 04 04:00:51.245157 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.245117 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdx8l\" (UniqueName: \"kubernetes.io/projected/a7f1c31b-8f96-4c56-ab05-8e22896d6533-kube-api-access-zdx8l\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.245339 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.245166 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-data\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.245339 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.245269 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-credential-socket\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.245339 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.245294 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-workload-certs\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.245490 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.245385 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istiod-ca-cert\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.245490 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.245427 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-envoy\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.245490 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.245460 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-podinfo\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.245602 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.245546 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-token\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.245654 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.245612 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-workload-socket\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.346407 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.346329 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-credential-socket\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.346407 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.346373 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-workload-certs\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.346751 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.346433 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istiod-ca-cert\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.346751 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.346475 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-envoy\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.346751 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.346501 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-podinfo\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.346751 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.346532 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-token\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.346751 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.346582 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-workload-socket\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.347062 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.346773 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-credential-socket\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.347062 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.346818 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-workload-certs\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.347062 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.346910 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdx8l\" (UniqueName: \"kubernetes.io/projected/a7f1c31b-8f96-4c56-ab05-8e22896d6533-kube-api-access-zdx8l\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.347062 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.346999 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-data\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.347278 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.347169 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-workload-socket\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.347336 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.347273 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istiod-ca-cert\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.347395 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.347360 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-data\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.349294 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.349267 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-podinfo\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.349530 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.349512 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-envoy\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.360891 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.360864 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-token\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.361054 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.361037 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdx8l\" (UniqueName: \"kubernetes.io/projected/a7f1c31b-8f96-4c56-ab05-8e22896d6533-kube-api-access-zdx8l\") pod \"e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.408269 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.408239 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:51.707619 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.707583 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl"] Nov 04 04:00:51.708732 ip-10-0-138-9 kubenswrapper[4966]: W1104 04:00:51.708701 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f1c31b_8f96_4c56_ab05_8e22896d6533.slice/crio-9c18db7b4d61fff7cea222a7c9ca4eb34898c64157a627afa3c118d016156a6d WatchSource:0}: Error finding container 9c18db7b4d61fff7cea222a7c9ca4eb34898c64157a627afa3c118d016156a6d: Status 404 returned error can't find the container with id 9c18db7b4d61fff7cea222a7c9ca4eb34898c64157a627afa3c118d016156a6d Nov 04 04:00:51.711675 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.711634 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:00:51.711776 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.711718 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:00:51.711776 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:51.711755 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:00:52.315634 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:52.315591 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" event={"ID":"a7f1c31b-8f96-4c56-ab05-8e22896d6533","Type":"ContainerStarted","Data":"690639b0b4db94b94d5d56158a9fcd34b4c7c8f6f34d792764c4bc985dc0ae29"} Nov 04 04:00:52.316094 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:52.315659 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" event={"ID":"a7f1c31b-8f96-4c56-ab05-8e22896d6533","Type":"ContainerStarted","Data":"9c18db7b4d61fff7cea222a7c9ca4eb34898c64157a627afa3c118d016156a6d"} Nov 04 04:00:52.344008 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:52.343966 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" podStartSLOduration=1.343954203 podStartE2EDuration="1.343954203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:00:52.342629479 +0000 UTC m=+696.143575297" watchObservedRunningTime="2026-11-04 04:00:52.343954203 +0000 UTC m=+696.144900011" Nov 04 04:00:52.408738 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:52.408700 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:52.410467 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:52.410445 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 04 04:00:52.410586 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:52.410490 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" podUID="a7f1c31b-8f96-4c56-ab05-8e22896d6533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 04 04:00:53.105698 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:53.105649 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw"] Nov 04 04:00:53.408757 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:53.408676 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 04 04:00:53.409145 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:53.408750 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" podUID="a7f1c31b-8f96-4c56-ab05-8e22896d6533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 04 04:00:53.829151 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:53.829122 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8"] Nov 04 04:00:53.833869 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:53.833841 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:53.851998 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:53.851898 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4afe0e\"/\"distinct-w0-4afe0e-kserve-self-signed-certs\"" Nov 04 04:00:53.852137 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:53.852064 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4afe0e\"/\"default-dockercfg-gqkhp\"" Nov 04 04:00:53.873944 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:53.870358 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4afe0e\"/\"kube-root-ca.crt\"" Nov 04 04:00:53.873944 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:53.870475 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4afe0e\"/\"openshift-service-ca.crt\"" Nov 04 04:00:53.885979 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:53.885890 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8"] Nov 04 04:00:54.000801 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.000758 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hkrx\" (UniqueName: \"kubernetes.io/projected/76911349-4357-4a41-9218-a6077aed29bc-kube-api-access-6hkrx\") pod \"distinct-w0-4afe0e-kserve-5f94d449bd-blnp8\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:54.001019 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.000819 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-dshm\") pod \"distinct-w0-4afe0e-kserve-5f94d449bd-blnp8\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:54.001019 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.000887 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-home\") pod \"distinct-w0-4afe0e-kserve-5f94d449bd-blnp8\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:54.001019 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.000938 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-tmp-dir\") pod \"distinct-w0-4afe0e-kserve-5f94d449bd-blnp8\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:54.001019 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.000986 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-model-cache\") pod \"distinct-w0-4afe0e-kserve-5f94d449bd-blnp8\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:54.001264 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.001106 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76911349-4357-4a41-9218-a6077aed29bc-tls-certs\") pod \"distinct-w0-4afe0e-kserve-5f94d449bd-blnp8\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:54.103220 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.102755 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-model-cache\") pod \"distinct-w0-4afe0e-kserve-5f94d449bd-blnp8\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:54.103393 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.103295 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76911349-4357-4a41-9218-a6077aed29bc-tls-certs\") pod \"distinct-w0-4afe0e-kserve-5f94d449bd-blnp8\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:54.103458 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.103439 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hkrx\" (UniqueName: \"kubernetes.io/projected/76911349-4357-4a41-9218-a6077aed29bc-kube-api-access-6hkrx\") pod \"distinct-w0-4afe0e-kserve-5f94d449bd-blnp8\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:54.103514 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.103474 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-dshm\") pod \"distinct-w0-4afe0e-kserve-5f94d449bd-blnp8\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:54.103568 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.103545 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-home\") pod \"distinct-w0-4afe0e-kserve-5f94d449bd-blnp8\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:54.103618 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.103569 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-tmp-dir\") pod \"distinct-w0-4afe0e-kserve-5f94d449bd-blnp8\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:54.103818 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.103793 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-model-cache\") pod \"distinct-w0-4afe0e-kserve-5f94d449bd-blnp8\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:54.103997 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.103973 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-tmp-dir\") pod \"distinct-w0-4afe0e-kserve-5f94d449bd-blnp8\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:54.104143 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.104120 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-home\") pod \"distinct-w0-4afe0e-kserve-5f94d449bd-blnp8\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:54.105826 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.105775 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-dshm\") pod \"distinct-w0-4afe0e-kserve-5f94d449bd-blnp8\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:54.106177 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.106157 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76911349-4357-4a41-9218-a6077aed29bc-tls-certs\") pod \"distinct-w0-4afe0e-kserve-5f94d449bd-blnp8\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:54.115995 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.115969 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hkrx\" (UniqueName: \"kubernetes.io/projected/76911349-4357-4a41-9218-a6077aed29bc-kube-api-access-6hkrx\") pod \"distinct-w0-4afe0e-kserve-5f94d449bd-blnp8\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:54.149779 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.149745 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:54.408752 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.408667 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 04 04:00:54.408752 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.408725 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" podUID="a7f1c31b-8f96-4c56-ab05-8e22896d6533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 04 04:00:54.486573 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:54.486538 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8"] Nov 04 04:00:54.486956 ip-10-0-138-9 kubenswrapper[4966]: W1104 04:00:54.486909 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76911349_4357_4a41_9218_a6077aed29bc.slice/crio-ea02093753988cd232bce5b4623b9bef6112a8c3ed0ee2acd22723ae9e2ac1d8 WatchSource:0}: Error finding container ea02093753988cd232bce5b4623b9bef6112a8c3ed0ee2acd22723ae9e2ac1d8: Status 404 returned error can't find the container with id ea02093753988cd232bce5b4623b9bef6112a8c3ed0ee2acd22723ae9e2ac1d8 Nov 04 04:00:55.339870 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:55.339834 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" event={"ID":"76911349-4357-4a41-9218-a6077aed29bc","Type":"ContainerStarted","Data":"41f19ff74a4c8cc49968fa9cb6ac1473c42345e7b091695e16670d85bd629daa"} Nov 04 04:00:55.340073 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:55.339878 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" event={"ID":"76911349-4357-4a41-9218-a6077aed29bc","Type":"ContainerStarted","Data":"ea02093753988cd232bce5b4623b9bef6112a8c3ed0ee2acd22723ae9e2ac1d8"} Nov 04 04:00:55.377941 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:55.377416 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" podStartSLOduration=2.377393886 podStartE2EDuration="2.377393886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:00:55.376342945 +0000 UTC m=+699.177288754" watchObservedRunningTime="2026-11-04 04:00:55.377393886 +0000 UTC m=+699.178339695" Nov 04 04:00:55.408903 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:55.408869 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 04 04:00:55.409313 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:55.408950 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" podUID="a7f1c31b-8f96-4c56-ab05-8e22896d6533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 04 04:00:56.150196 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:56.150164 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:00:56.409400 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:56.409321 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 04 04:00:56.409400 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:56.409380 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" podUID="a7f1c31b-8f96-4c56-ab05-8e22896d6533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 04 04:00:57.409294 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:57.409254 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 04 04:00:57.409455 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:57.409320 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" podUID="a7f1c31b-8f96-4c56-ab05-8e22896d6533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 04 04:00:58.633582 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:58.633541 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:58.634029 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:58.634016 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:00:58.635013 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:00:58.634990 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:01:03.673480 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:03.673442 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn"] Nov 04 04:01:03.684449 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:03.684414 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs"] Nov 04 04:01:04.150445 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:04.150413 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:01:04.163681 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:04.163646 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:01:04.406439 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:04.406364 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:01:05.679166 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:01:05.679121 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 04:01:07.078282 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:07.078238 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg"] Nov 04 04:01:08.342105 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.342031 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" podUID="6646d4de-0e96-4e6d-91f9-a426a7a862b4" containerName="main" containerID="cri-o://fd574be7a86fb72def83728d94322c91f226444712c0b48c586c45646b6bbac7" gracePeriod=45 Nov 04 04:01:08.619228 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.619205 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 04:01:08.675954 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.675878 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn" podUID="61c9dc55-1b5a-4bcc-94de-7b3eaa030b62" containerName="payload-processing" containerID="cri-o://c07477dbb920bf4e1bee79da495c060dc6ef12482f968bea5964b72f7c109e01" gracePeriod=25 Nov 04 04:01:08.684179 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.684140 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-dshm\") pod \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " Nov 04 04:01:08.684313 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.684235 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-tmp-dir\") pod \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " Nov 04 04:01:08.684388 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.684317 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-model-cache\") pod \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " Nov 04 04:01:08.684448 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.684415 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-home\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-home\") pod \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " Nov 04 04:01:08.684506 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.684493 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6646d4de-0e96-4e6d-91f9-a426a7a862b4-kube-api-access-789qg\" (UniqueName: \"kubernetes.io/projected/6646d4de-0e96-4e6d-91f9-a426a7a862b4-kube-api-access-789qg\") pod \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " Nov 04 04:01:08.684604 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.684587 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6646d4de-0e96-4e6d-91f9-a426a7a862b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/6646d4de-0e96-4e6d-91f9-a426a7a862b4-tls-certs\") pod \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\" (UID: \"6646d4de-0e96-4e6d-91f9-a426a7a862b4\") " Nov 04 04:01:08.684671 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.684647 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-model-cache" pod "6646d4de-0e96-4e6d-91f9-a426a7a862b4" (UID: "6646d4de-0e96-4e6d-91f9-a426a7a862b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:08.684740 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.684720 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-home" pod "6646d4de-0e96-4e6d-91f9-a426a7a862b4" (UID: "6646d4de-0e96-4e6d-91f9-a426a7a862b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:08.685653 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.685004 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs" podUID="08fb1e7a-0bdb-4529-a421-345600882d98" containerName="payload-pre-processing" containerID="cri-o://166c42e26fbc231f139521ade0a1cfc6b3aa9588c8d4a7c712fe5297d3658fb7" gracePeriod=25 Nov 04 04:01:08.685653 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.685115 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-tmp-dir" pod "6646d4de-0e96-4e6d-91f9-a426a7a862b4" (UID: "6646d4de-0e96-4e6d-91f9-a426a7a862b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:08.685653 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.685043 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-model-cache\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:08.685653 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.685167 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-home\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:08.688552 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.688530 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-dshm" pod "6646d4de-0e96-4e6d-91f9-a426a7a862b4" (UID: "6646d4de-0e96-4e6d-91f9-a426a7a862b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:08.688665 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.688631 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6646d4de-0e96-4e6d-91f9-a426a7a862b4-kube-api-access-789qg" pod "6646d4de-0e96-4e6d-91f9-a426a7a862b4" (UID: "6646d4de-0e96-4e6d-91f9-a426a7a862b4"). InnerVolumeSpecName "kube-api-access-789qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:01:08.692985 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.692960 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6646d4de-0e96-4e6d-91f9-a426a7a862b4-tls-certs" pod "6646d4de-0e96-4e6d-91f9-a426a7a862b4" (UID: "6646d4de-0e96-4e6d-91f9-a426a7a862b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:01:08.786325 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.786297 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6646d4de-0e96-4e6d-91f9-a426a7a862b4-tls-certs\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:08.786325 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.786323 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-dshm\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:08.786490 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.786333 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6646d4de-0e96-4e6d-91f9-a426a7a862b4-tmp-dir\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:08.786490 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.786343 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-789qg\" (UniqueName: \"kubernetes.io/projected/6646d4de-0e96-4e6d-91f9-a426a7a862b4-kube-api-access-789qg\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:08.957749 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.957715 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn" Nov 04 04:01:08.996219 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:08.996193 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs" Nov 04 04:01:09.090142 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.090106 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c9dc55-1b5a-4bcc-94de-7b3eaa030b62-kube-api-access-m2q6p\" (UniqueName: \"kubernetes.io/projected/61c9dc55-1b5a-4bcc-94de-7b3eaa030b62-kube-api-access-m2q6p\") pod \"61c9dc55-1b5a-4bcc-94de-7b3eaa030b62\" (UID: \"61c9dc55-1b5a-4bcc-94de-7b3eaa030b62\") " Nov 04 04:01:09.090335 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.090267 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61c9dc55-1b5a-4bcc-94de-7b3eaa030b62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c9dc55-1b5a-4bcc-94de-7b3eaa030b62-plugins-config-volume\") pod \"61c9dc55-1b5a-4bcc-94de-7b3eaa030b62\" (UID: \"61c9dc55-1b5a-4bcc-94de-7b3eaa030b62\") " Nov 04 04:01:09.090335 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.090330 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08fb1e7a-0bdb-4529-a421-345600882d98-kube-api-access-2tvqt\" (UniqueName: \"kubernetes.io/projected/08fb1e7a-0bdb-4529-a421-345600882d98-kube-api-access-2tvqt\") pod \"08fb1e7a-0bdb-4529-a421-345600882d98\" (UID: \"08fb1e7a-0bdb-4529-a421-345600882d98\") " Nov 04 04:01:09.090449 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.090363 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08fb1e7a-0bdb-4529-a421-345600882d98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08fb1e7a-0bdb-4529-a421-345600882d98-plugins-config-volume\") pod \"08fb1e7a-0bdb-4529-a421-345600882d98\" (UID: \"08fb1e7a-0bdb-4529-a421-345600882d98\") " Nov 04 04:01:09.090832 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.090796 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c9dc55-1b5a-4bcc-94de-7b3eaa030b62-plugins-config-volume" pod "61c9dc55-1b5a-4bcc-94de-7b3eaa030b62" (UID: "61c9dc55-1b5a-4bcc-94de-7b3eaa030b62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:01:09.091007 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.090866 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08fb1e7a-0bdb-4529-a421-345600882d98-plugins-config-volume" pod "08fb1e7a-0bdb-4529-a421-345600882d98" (UID: "08fb1e7a-0bdb-4529-a421-345600882d98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:01:09.092487 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.092459 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c9dc55-1b5a-4bcc-94de-7b3eaa030b62-kube-api-access-m2q6p" pod "61c9dc55-1b5a-4bcc-94de-7b3eaa030b62" (UID: "61c9dc55-1b5a-4bcc-94de-7b3eaa030b62"). InnerVolumeSpecName "kube-api-access-m2q6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:01:09.092587 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.092485 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08fb1e7a-0bdb-4529-a421-345600882d98-kube-api-access-2tvqt" pod "08fb1e7a-0bdb-4529-a421-345600882d98" (UID: "08fb1e7a-0bdb-4529-a421-345600882d98"). InnerVolumeSpecName "kube-api-access-2tvqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:01:09.192112 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.192026 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c9dc55-1b5a-4bcc-94de-7b3eaa030b62-plugins-config-volume\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:09.192112 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.192056 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tvqt\" (UniqueName: \"kubernetes.io/projected/08fb1e7a-0bdb-4529-a421-345600882d98-kube-api-access-2tvqt\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:09.192112 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.192066 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08fb1e7a-0bdb-4529-a421-345600882d98-plugins-config-volume\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:09.192112 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.192076 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2q6p\" (UniqueName: \"kubernetes.io/projected/61c9dc55-1b5a-4bcc-94de-7b3eaa030b62-kube-api-access-m2q6p\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:09.429580 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.429546 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7"] Nov 04 04:01:09.430209 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.430192 4966 cpu_manager.go:414] "containerMap: removing container" podUID="08fb1e7a-0bdb-4529-a421-345600882d98" containerName="payload-pre-processing" Nov 04 04:01:09.430291 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.430209 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08fb1e7a-0bdb-4529-a421-345600882d98" containerName="payload-pre-processing" Nov 04 04:01:09.430291 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.430224 4966 cpu_manager.go:414] "containerMap: removing container" podUID="61c9dc55-1b5a-4bcc-94de-7b3eaa030b62" containerName="payload-processing" Nov 04 04:01:09.430291 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.430233 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c9dc55-1b5a-4bcc-94de-7b3eaa030b62" containerName="payload-processing" Nov 04 04:01:09.430291 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.430254 4966 cpu_manager.go:414] "containerMap: removing container" podUID="6646d4de-0e96-4e6d-91f9-a426a7a862b4" containerName="main" Nov 04 04:01:09.430291 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.430259 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6646d4de-0e96-4e6d-91f9-a426a7a862b4" containerName="main" Nov 04 04:01:09.430452 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.430348 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="08fb1e7a-0bdb-4529-a421-345600882d98" containerName="payload-pre-processing" Nov 04 04:01:09.430452 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.430358 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c9dc55-1b5a-4bcc-94de-7b3eaa030b62" containerName="payload-processing" Nov 04 04:01:09.430452 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.430366 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6646d4de-0e96-4e6d-91f9-a426a7a862b4" containerName="main" Nov 04 04:01:09.437590 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.437559 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:09.443401 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.443319 4966 generic.go:356] "Generic (PLEG): container finished" podID="08fb1e7a-0bdb-4529-a421-345600882d98" containerID="166c42e26fbc231f139521ade0a1cfc6b3aa9588c8d4a7c712fe5297d3658fb7" exitCode=0 Nov 04 04:01:09.443401 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.443362 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs" Nov 04 04:01:09.443571 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.443397 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs" event={"ID":"08fb1e7a-0bdb-4529-a421-345600882d98","Type":"ContainerDied","Data":"166c42e26fbc231f139521ade0a1cfc6b3aa9588c8d4a7c712fe5297d3658fb7"} Nov 04 04:01:09.443571 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.443449 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs" event={"ID":"08fb1e7a-0bdb-4529-a421-345600882d98","Type":"ContainerDied","Data":"182de54b6d81ec2280a7fd15b98e0c17ddedbe1fce446423ff83b1742a21f3d9"} Nov 04 04:01:09.443571 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.443477 4966 scope.go:122] "RemoveContainer" containerID="166c42e26fbc231f139521ade0a1cfc6b3aa9588c8d4a7c712fe5297d3658fb7" Nov 04 04:01:09.443807 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.443688 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7"] Nov 04 04:01:09.449176 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.449149 4966 generic.go:356] "Generic (PLEG): container finished" podID="61c9dc55-1b5a-4bcc-94de-7b3eaa030b62" containerID="c07477dbb920bf4e1bee79da495c060dc6ef12482f968bea5964b72f7c109e01" exitCode=0 Nov 04 04:01:09.449311 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.449273 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn" event={"ID":"61c9dc55-1b5a-4bcc-94de-7b3eaa030b62","Type":"ContainerDied","Data":"c07477dbb920bf4e1bee79da495c060dc6ef12482f968bea5964b72f7c109e01"} Nov 04 04:01:09.449415 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.449317 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn" event={"ID":"61c9dc55-1b5a-4bcc-94de-7b3eaa030b62","Type":"ContainerDied","Data":"c13d967f30ac047b9deeb948e450c4e6b88bbaeb0ef98b5c76061548348d1572"} Nov 04 04:01:09.449415 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.449400 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn" Nov 04 04:01:09.453181 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.450842 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-64c20d\"/\"distinct-2-w4-64c20d-kserve-self-signed-certs\"" Nov 04 04:01:09.453181 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.451633 4966 generic.go:356] "Generic (PLEG): container finished" podID="6646d4de-0e96-4e6d-91f9-a426a7a862b4" containerID="fd574be7a86fb72def83728d94322c91f226444712c0b48c586c45646b6bbac7" exitCode=0 Nov 04 04:01:09.453181 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.451667 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" Nov 04 04:01:09.453181 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.451675 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" event={"ID":"6646d4de-0e96-4e6d-91f9-a426a7a862b4","Type":"ContainerDied","Data":"fd574be7a86fb72def83728d94322c91f226444712c0b48c586c45646b6bbac7"} Nov 04 04:01:09.453181 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.451702 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw" event={"ID":"6646d4de-0e96-4e6d-91f9-a426a7a862b4","Type":"ContainerDied","Data":"2c612fd7a82b989785339debf7ba7b981fb3616a061472af088a01791ba7608f"} Nov 04 04:01:09.471679 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.470958 4966 scope.go:122] "RemoveContainer" containerID="166c42e26fbc231f139521ade0a1cfc6b3aa9588c8d4a7c712fe5297d3658fb7" Nov 04 04:01:09.471679 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:01:09.471404 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"166c42e26fbc231f139521ade0a1cfc6b3aa9588c8d4a7c712fe5297d3658fb7\": container with ID starting with 166c42e26fbc231f139521ade0a1cfc6b3aa9588c8d4a7c712fe5297d3658fb7 not found: ID does not exist" containerID="166c42e26fbc231f139521ade0a1cfc6b3aa9588c8d4a7c712fe5297d3658fb7" Nov 04 04:01:09.471679 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.471448 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"166c42e26fbc231f139521ade0a1cfc6b3aa9588c8d4a7c712fe5297d3658fb7"} err="failed to get container status \"166c42e26fbc231f139521ade0a1cfc6b3aa9588c8d4a7c712fe5297d3658fb7\": rpc error: code = NotFound desc = could not find container \"166c42e26fbc231f139521ade0a1cfc6b3aa9588c8d4a7c712fe5297d3658fb7\": container with ID starting with 166c42e26fbc231f139521ade0a1cfc6b3aa9588c8d4a7c712fe5297d3658fb7 not found: ID does not exist" Nov 04 04:01:09.471679 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.471475 4966 scope.go:122] "RemoveContainer" containerID="c07477dbb920bf4e1bee79da495c060dc6ef12482f968bea5964b72f7c109e01" Nov 04 04:01:09.489148 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.489121 4966 scope.go:122] "RemoveContainer" containerID="c07477dbb920bf4e1bee79da495c060dc6ef12482f968bea5964b72f7c109e01" Nov 04 04:01:09.489442 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:01:09.489421 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07477dbb920bf4e1bee79da495c060dc6ef12482f968bea5964b72f7c109e01\": container with ID starting with c07477dbb920bf4e1bee79da495c060dc6ef12482f968bea5964b72f7c109e01 not found: ID does not exist" containerID="c07477dbb920bf4e1bee79da495c060dc6ef12482f968bea5964b72f7c109e01" Nov 04 04:01:09.489499 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.489449 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07477dbb920bf4e1bee79da495c060dc6ef12482f968bea5964b72f7c109e01"} err="failed to get container status \"c07477dbb920bf4e1bee79da495c060dc6ef12482f968bea5964b72f7c109e01\": rpc error: code = NotFound desc = could not find container \"c07477dbb920bf4e1bee79da495c060dc6ef12482f968bea5964b72f7c109e01\": container with ID starting with c07477dbb920bf4e1bee79da495c060dc6ef12482f968bea5964b72f7c109e01 not found: ID does not exist" Nov 04 04:01:09.489499 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.489466 4966 scope.go:122] "RemoveContainer" containerID="fd574be7a86fb72def83728d94322c91f226444712c0b48c586c45646b6bbac7" Nov 04 04:01:09.495941 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.495899 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-dshm\") pod \"distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:09.496058 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.496011 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-model-cache\") pod \"distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:09.496164 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.496139 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szbcp\" (UniqueName: \"kubernetes.io/projected/ba949fa3-73d8-4ae0-b586-b001b8988986-kube-api-access-szbcp\") pod \"distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:09.496278 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.496260 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-tmp-dir\") pod \"distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:09.496393 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.496291 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-home\") pod \"distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:09.496455 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.496427 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba949fa3-73d8-4ae0-b586-b001b8988986-tls-certs\") pod \"distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:09.498579 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.498558 4966 scope.go:122] "RemoveContainer" containerID="fd574be7a86fb72def83728d94322c91f226444712c0b48c586c45646b6bbac7" Nov 04 04:01:09.498871 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:01:09.498850 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd574be7a86fb72def83728d94322c91f226444712c0b48c586c45646b6bbac7\": container with ID starting with fd574be7a86fb72def83728d94322c91f226444712c0b48c586c45646b6bbac7 not found: ID does not exist" containerID="fd574be7a86fb72def83728d94322c91f226444712c0b48c586c45646b6bbac7" Nov 04 04:01:09.498989 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.498879 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd574be7a86fb72def83728d94322c91f226444712c0b48c586c45646b6bbac7"} err="failed to get container status \"fd574be7a86fb72def83728d94322c91f226444712c0b48c586c45646b6bbac7\": rpc error: code = NotFound desc = could not find container \"fd574be7a86fb72def83728d94322c91f226444712c0b48c586c45646b6bbac7\": container with ID starting with fd574be7a86fb72def83728d94322c91f226444712c0b48c586c45646b6bbac7 not found: ID does not exist" Nov 04 04:01:09.504795 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.504769 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn"] Nov 04 04:01:09.509036 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.509015 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn"] Nov 04 04:01:09.521057 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.521038 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs"] Nov 04 04:01:09.525552 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.525533 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs"] Nov 04 04:01:09.536693 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.536668 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw"] Nov 04 04:01:09.540700 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.540677 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-998997/premium-w3-998997-kserve-5d64f845fb-pt4gw"] Nov 04 04:01:09.597740 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.597702 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-model-cache\") pod \"distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:09.597967 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.597774 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szbcp\" (UniqueName: \"kubernetes.io/projected/ba949fa3-73d8-4ae0-b586-b001b8988986-kube-api-access-szbcp\") pod \"distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:09.597967 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.597831 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-tmp-dir\") pod \"distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:09.597967 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.597855 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-home\") pod \"distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:09.597967 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.597898 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba949fa3-73d8-4ae0-b586-b001b8988986-tls-certs\") pod \"distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:09.597967 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.597964 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-dshm\") pod \"distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:09.598263 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.598240 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-model-cache\") pod \"distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:09.598320 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.598282 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-tmp-dir\") pod \"distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:09.598368 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.598352 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-home\") pod \"distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:09.600572 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.600544 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-dshm\") pod \"distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:09.600723 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.600615 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba949fa3-73d8-4ae0-b586-b001b8988986-tls-certs\") pod \"distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:09.610824 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.610791 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szbcp\" (UniqueName: \"kubernetes.io/projected/ba949fa3-73d8-4ae0-b586-b001b8988986-kube-api-access-szbcp\") pod \"distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:09.767746 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:09.767666 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:10.054975 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:10.054944 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7"] Nov 04 04:01:10.056137 ip-10-0-138-9 kubenswrapper[4966]: W1104 04:01:10.056105 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba949fa3_73d8_4ae0_b586_b001b8988986.slice/crio-9642bf1a29d64756b0d5ca15cbb656176ca0f2b60d146106a9dc4d07919a8960 WatchSource:0}: Error finding container 9642bf1a29d64756b0d5ca15cbb656176ca0f2b60d146106a9dc4d07919a8960: Status 404 returned error can't find the container with id 9642bf1a29d64756b0d5ca15cbb656176ca0f2b60d146106a9dc4d07919a8960 Nov 04 04:01:10.342752 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:10.342707 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08fb1e7a-0bdb-4529-a421-345600882d98" path="/var/lib/kubelet/pods/08fb1e7a-0bdb-4529-a421-345600882d98/volumes" Nov 04 04:01:10.343255 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:10.343233 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c9dc55-1b5a-4bcc-94de-7b3eaa030b62" path="/var/lib/kubelet/pods/61c9dc55-1b5a-4bcc-94de-7b3eaa030b62/volumes" Nov 04 04:01:10.343722 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:10.343702 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6646d4de-0e96-4e6d-91f9-a426a7a862b4" path="/var/lib/kubelet/pods/6646d4de-0e96-4e6d-91f9-a426a7a862b4/volumes" Nov 04 04:01:10.463719 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:10.463677 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" event={"ID":"ba949fa3-73d8-4ae0-b586-b001b8988986","Type":"ContainerStarted","Data":"258fc19481ee4d7f679307d12ecfb252055323661032b8ecaacf365987b9c478"} Nov 04 04:01:10.464228 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:10.463733 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" event={"ID":"ba949fa3-73d8-4ae0-b586-b001b8988986","Type":"ContainerStarted","Data":"9642bf1a29d64756b0d5ca15cbb656176ca0f2b60d146106a9dc4d07919a8960"} Nov 04 04:01:10.488288 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:10.488226 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" podStartSLOduration=1.488209355 podStartE2EDuration="1.488209355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:01:10.487320638 +0000 UTC m=+714.288266451" watchObservedRunningTime="2026-11-04 04:01:10.488209355 +0000 UTC m=+714.289155163" Nov 04 04:01:10.675151 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:10.675071 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2"] Nov 04 04:01:10.768401 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:10.768372 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:11.722307 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:11.722265 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7"] Nov 04 04:01:12.078994 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:12.078955 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" podUID="e8517352-f4dc-4e65-8ac7-ccb0b01d5255" containerName="payload-processing" containerID="cri-o://398b12ee0e5eceb08f69a5e4482c133a7a718d9dce147060980cf364bf6b2228" gracePeriod=25 Nov 04 04:01:15.148261 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.148223 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62"] Nov 04 04:01:15.148700 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.148504 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" podUID="e3324a6f-e5f3-4ea0-9e43-e2900ae045f1" containerName="maas-api" containerID="cri-o://5ff811fc04260d118de37275abef1a949f2a6b8c706404e66bd6302440726b7c" gracePeriod=30 Nov 04 04:01:15.441950 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.441906 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" Nov 04 04:01:15.506461 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.506414 4966 generic.go:356] "Generic (PLEG): container finished" podID="e3324a6f-e5f3-4ea0-9e43-e2900ae045f1" containerID="5ff811fc04260d118de37275abef1a949f2a6b8c706404e66bd6302440726b7c" exitCode=0 Nov 04 04:01:15.506887 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.506481 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" Nov 04 04:01:15.506887 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.506501 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" event={"ID":"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1","Type":"ContainerDied","Data":"5ff811fc04260d118de37275abef1a949f2a6b8c706404e66bd6302440726b7c"} Nov 04 04:01:15.506887 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.506567 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" event={"ID":"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1","Type":"ContainerDied","Data":"6f04410bdbfd319b6628d2745ae17811dadaa43f024f0fa4e25b5c7c84400ad5"} Nov 04 04:01:15.506887 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.506597 4966 scope.go:122] "RemoveContainer" containerID="5ff811fc04260d118de37275abef1a949f2a6b8c706404e66bd6302440726b7c" Nov 04 04:01:15.517238 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.517211 4966 scope.go:122] "RemoveContainer" containerID="5ff811fc04260d118de37275abef1a949f2a6b8c706404e66bd6302440726b7c" Nov 04 04:01:15.517538 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:01:15.517515 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff811fc04260d118de37275abef1a949f2a6b8c706404e66bd6302440726b7c\": container with ID starting with 5ff811fc04260d118de37275abef1a949f2a6b8c706404e66bd6302440726b7c not found: ID does not exist" containerID="5ff811fc04260d118de37275abef1a949f2a6b8c706404e66bd6302440726b7c" Nov 04 04:01:15.517587 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.517546 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff811fc04260d118de37275abef1a949f2a6b8c706404e66bd6302440726b7c"} err="failed to get container status \"5ff811fc04260d118de37275abef1a949f2a6b8c706404e66bd6302440726b7c\": rpc error: code = NotFound desc = could not find container \"5ff811fc04260d118de37275abef1a949f2a6b8c706404e66bd6302440726b7c\": container with ID starting with 5ff811fc04260d118de37275abef1a949f2a6b8c706404e66bd6302440726b7c not found: ID does not exist" Nov 04 04:01:15.591024 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.590992 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-kube-api-access-tzgw2\" (UniqueName: \"kubernetes.io/projected/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-kube-api-access-tzgw2\") pod \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\" (UID: \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\") " Nov 04 04:01:15.591209 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.591086 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-openshift-service-ca\") pod \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\" (UID: \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\") " Nov 04 04:01:15.591209 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.591178 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-metrics-certs\") pod \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\" (UID: \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\") " Nov 04 04:01:15.591312 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.591296 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-maas-api-tls\") pod \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\" (UID: \"e3324a6f-e5f3-4ea0-9e43-e2900ae045f1\") " Nov 04 04:01:15.591434 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.591403 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-openshift-service-ca" pod "e3324a6f-e5f3-4ea0-9e43-e2900ae045f1" (UID: "e3324a6f-e5f3-4ea0-9e43-e2900ae045f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:01:15.591740 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.591718 4966 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-openshift-service-ca\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:15.593455 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.593433 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-kube-api-access-tzgw2" pod "e3324a6f-e5f3-4ea0-9e43-e2900ae045f1" (UID: "e3324a6f-e5f3-4ea0-9e43-e2900ae045f1"). InnerVolumeSpecName "kube-api-access-tzgw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:01:15.593721 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.593701 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-maas-api-tls" pod "e3324a6f-e5f3-4ea0-9e43-e2900ae045f1" (UID: "e3324a6f-e5f3-4ea0-9e43-e2900ae045f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:01:15.593783 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.593722 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-metrics-certs" pod "e3324a6f-e5f3-4ea0-9e43-e2900ae045f1" (UID: "e3324a6f-e5f3-4ea0-9e43-e2900ae045f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:01:15.676269 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.676168 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2" podUID="a85861de-cb4f-4da6-ad76-4e55e957d797" containerName="payload-processing" containerID="cri-o://6094f0065d70402d394fd64fa344625643a355418831a0ba651dfeb6d3703ab5" gracePeriod=25 Nov 04 04:01:15.693316 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.693288 4966 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-metrics-certs\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:15.693423 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.693320 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-maas-api-tls\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:15.693423 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.693334 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzgw2\" (UniqueName: \"kubernetes.io/projected/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1-kube-api-access-tzgw2\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:15.837286 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.832563 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62"] Nov 04 04:01:15.837286 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.837232 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62"] Nov 04 04:01:15.929386 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:15.929316 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2" Nov 04 04:01:16.097776 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:16.097741 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85861de-cb4f-4da6-ad76-4e55e957d797-kube-api-access-sj8xd\" (UniqueName: \"kubernetes.io/projected/a85861de-cb4f-4da6-ad76-4e55e957d797-kube-api-access-sj8xd\") pod \"a85861de-cb4f-4da6-ad76-4e55e957d797\" (UID: \"a85861de-cb4f-4da6-ad76-4e55e957d797\") " Nov 04 04:01:16.098006 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:16.097968 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a85861de-cb4f-4da6-ad76-4e55e957d797-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a85861de-cb4f-4da6-ad76-4e55e957d797-plugins-config-volume\") pod \"a85861de-cb4f-4da6-ad76-4e55e957d797\" (UID: \"a85861de-cb4f-4da6-ad76-4e55e957d797\") " Nov 04 04:01:16.098404 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:16.098379 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85861de-cb4f-4da6-ad76-4e55e957d797-plugins-config-volume" pod "a85861de-cb4f-4da6-ad76-4e55e957d797" (UID: "a85861de-cb4f-4da6-ad76-4e55e957d797"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:01:16.099987 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:16.099963 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85861de-cb4f-4da6-ad76-4e55e957d797-kube-api-access-sj8xd" pod "a85861de-cb4f-4da6-ad76-4e55e957d797" (UID: "a85861de-cb4f-4da6-ad76-4e55e957d797"). InnerVolumeSpecName "kube-api-access-sj8xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:01:16.199676 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:16.199604 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a85861de-cb4f-4da6-ad76-4e55e957d797-plugins-config-volume\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:16.199676 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:16.199630 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj8xd\" (UniqueName: \"kubernetes.io/projected/a85861de-cb4f-4da6-ad76-4e55e957d797-kube-api-access-sj8xd\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:16.342849 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:16.342816 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3324a6f-e5f3-4ea0-9e43-e2900ae045f1" path="/var/lib/kubelet/pods/e3324a6f-e5f3-4ea0-9e43-e2900ae045f1/volumes" Nov 04 04:01:16.513573 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:16.513539 4966 generic.go:356] "Generic (PLEG): container finished" podID="a85861de-cb4f-4da6-ad76-4e55e957d797" containerID="6094f0065d70402d394fd64fa344625643a355418831a0ba651dfeb6d3703ab5" exitCode=0 Nov 04 04:01:16.513716 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:16.513613 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2" Nov 04 04:01:16.513716 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:16.513626 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2" event={"ID":"a85861de-cb4f-4da6-ad76-4e55e957d797","Type":"ContainerDied","Data":"6094f0065d70402d394fd64fa344625643a355418831a0ba651dfeb6d3703ab5"} Nov 04 04:01:16.513716 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:16.513701 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2" event={"ID":"a85861de-cb4f-4da6-ad76-4e55e957d797","Type":"ContainerDied","Data":"ef4ea222d36b879af0700155309672721d3e135fd9d042e323b4f1bb754a7f46"} Nov 04 04:01:16.513901 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:16.513725 4966 scope.go:122] "RemoveContainer" containerID="6094f0065d70402d394fd64fa344625643a355418831a0ba651dfeb6d3703ab5" Nov 04 04:01:16.525949 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:16.525928 4966 scope.go:122] "RemoveContainer" containerID="6094f0065d70402d394fd64fa344625643a355418831a0ba651dfeb6d3703ab5" Nov 04 04:01:16.526231 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:01:16.526212 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6094f0065d70402d394fd64fa344625643a355418831a0ba651dfeb6d3703ab5\": container with ID starting with 6094f0065d70402d394fd64fa344625643a355418831a0ba651dfeb6d3703ab5 not found: ID does not exist" containerID="6094f0065d70402d394fd64fa344625643a355418831a0ba651dfeb6d3703ab5" Nov 04 04:01:16.526292 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:16.526241 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6094f0065d70402d394fd64fa344625643a355418831a0ba651dfeb6d3703ab5"} err="failed to get container status \"6094f0065d70402d394fd64fa344625643a355418831a0ba651dfeb6d3703ab5\": rpc error: code = NotFound desc = could not find container \"6094f0065d70402d394fd64fa344625643a355418831a0ba651dfeb6d3703ab5\": container with ID starting with 6094f0065d70402d394fd64fa344625643a355418831a0ba651dfeb6d3703ab5 not found: ID does not exist" Nov 04 04:01:16.537165 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:16.537133 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2"] Nov 04 04:01:16.540804 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:16.540785 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2"] Nov 04 04:01:16.723156 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:16.723066 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7" podUID="6d5ce632-2597-47ef-bb2f-dd41f330c12f" containerName="payload-pre-processing" containerID="cri-o://ca31ca80e08b06efc1b1620cb5416a22a4c1b7b5adc4c9363b3f66d21fd065f4" gracePeriod=25 Nov 04 04:01:17.023177 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:17.023149 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7" Nov 04 04:01:17.109226 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:17.109197 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d5ce632-2597-47ef-bb2f-dd41f330c12f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5ce632-2597-47ef-bb2f-dd41f330c12f-plugins-config-volume\") pod \"6d5ce632-2597-47ef-bb2f-dd41f330c12f\" (UID: \"6d5ce632-2597-47ef-bb2f-dd41f330c12f\") " Nov 04 04:01:17.109406 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:17.109254 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d5ce632-2597-47ef-bb2f-dd41f330c12f-kube-api-access-rw95q\" (UniqueName: \"kubernetes.io/projected/6d5ce632-2597-47ef-bb2f-dd41f330c12f-kube-api-access-rw95q\") pod \"6d5ce632-2597-47ef-bb2f-dd41f330c12f\" (UID: \"6d5ce632-2597-47ef-bb2f-dd41f330c12f\") " Nov 04 04:01:17.109616 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:17.109590 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d5ce632-2597-47ef-bb2f-dd41f330c12f-plugins-config-volume" pod "6d5ce632-2597-47ef-bb2f-dd41f330c12f" (UID: "6d5ce632-2597-47ef-bb2f-dd41f330c12f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:01:17.111438 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:17.111412 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5ce632-2597-47ef-bb2f-dd41f330c12f-kube-api-access-rw95q" pod "6d5ce632-2597-47ef-bb2f-dd41f330c12f" (UID: "6d5ce632-2597-47ef-bb2f-dd41f330c12f"). InnerVolumeSpecName "kube-api-access-rw95q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:01:17.210775 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:17.210732 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5ce632-2597-47ef-bb2f-dd41f330c12f-plugins-config-volume\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:17.210775 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:17.210778 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw95q\" (UniqueName: \"kubernetes.io/projected/6d5ce632-2597-47ef-bb2f-dd41f330c12f-kube-api-access-rw95q\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:17.521521 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:17.521485 4966 generic.go:356] "Generic (PLEG): container finished" podID="6d5ce632-2597-47ef-bb2f-dd41f330c12f" containerID="ca31ca80e08b06efc1b1620cb5416a22a4c1b7b5adc4c9363b3f66d21fd065f4" exitCode=0 Nov 04 04:01:17.521732 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:17.521529 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7" Nov 04 04:01:17.521732 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:17.521570 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7" event={"ID":"6d5ce632-2597-47ef-bb2f-dd41f330c12f","Type":"ContainerDied","Data":"ca31ca80e08b06efc1b1620cb5416a22a4c1b7b5adc4c9363b3f66d21fd065f4"} Nov 04 04:01:17.521732 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:17.521628 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7" event={"ID":"6d5ce632-2597-47ef-bb2f-dd41f330c12f","Type":"ContainerDied","Data":"d6ca3d9cbfcd0dbe07e90948821167a43a76ae0901cae59308bae0fe08abe7e8"} Nov 04 04:01:17.521732 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:17.521657 4966 scope.go:122] "RemoveContainer" containerID="ca31ca80e08b06efc1b1620cb5416a22a4c1b7b5adc4c9363b3f66d21fd065f4" Nov 04 04:01:17.532267 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:17.532251 4966 scope.go:122] "RemoveContainer" containerID="ca31ca80e08b06efc1b1620cb5416a22a4c1b7b5adc4c9363b3f66d21fd065f4" Nov 04 04:01:17.532545 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:01:17.532526 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca31ca80e08b06efc1b1620cb5416a22a4c1b7b5adc4c9363b3f66d21fd065f4\": container with ID starting with ca31ca80e08b06efc1b1620cb5416a22a4c1b7b5adc4c9363b3f66d21fd065f4 not found: ID does not exist" containerID="ca31ca80e08b06efc1b1620cb5416a22a4c1b7b5adc4c9363b3f66d21fd065f4" Nov 04 04:01:17.532594 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:17.532554 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca31ca80e08b06efc1b1620cb5416a22a4c1b7b5adc4c9363b3f66d21fd065f4"} err="failed to get container status \"ca31ca80e08b06efc1b1620cb5416a22a4c1b7b5adc4c9363b3f66d21fd065f4\": rpc error: code = NotFound desc = could not find container \"ca31ca80e08b06efc1b1620cb5416a22a4c1b7b5adc4c9363b3f66d21fd065f4\": container with ID starting with ca31ca80e08b06efc1b1620cb5416a22a4c1b7b5adc4c9363b3f66d21fd065f4 not found: ID does not exist" Nov 04 04:01:17.545839 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:17.545815 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7"] Nov 04 04:01:17.548476 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:17.548435 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7"] Nov 04 04:01:18.340959 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:18.340908 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d5ce632-2597-47ef-bb2f-dd41f330c12f" path="/var/lib/kubelet/pods/6d5ce632-2597-47ef-bb2f-dd41f330c12f/volumes" Nov 04 04:01:18.341456 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:18.341436 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a85861de-cb4f-4da6-ad76-4e55e957d797" path="/var/lib/kubelet/pods/a85861de-cb4f-4da6-ad76-4e55e957d797/volumes" Nov 04 04:01:18.398165 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:18.398126 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-998997-54bf8648c7-4dh62" podUID="e3324a6f-e5f3-4ea0-9e43-e2900ae045f1" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.64:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 04:01:18.421247 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:18.421200 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" podUID="e8517352-f4dc-4e65-8ac7-ccb0b01d5255" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 04:01:19.768374 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:19.768333 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:19.781449 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:19.781422 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:20.586235 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.586203 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:20.760713 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.760671 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8"] Nov 04 04:01:20.761235 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.761218 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e3324a6f-e5f3-4ea0-9e43-e2900ae045f1" containerName="maas-api" Nov 04 04:01:20.761235 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.761235 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3324a6f-e5f3-4ea0-9e43-e2900ae045f1" containerName="maas-api" Nov 04 04:01:20.761375 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.761263 4966 cpu_manager.go:414] "containerMap: removing container" podUID="a85861de-cb4f-4da6-ad76-4e55e957d797" containerName="payload-processing" Nov 04 04:01:20.761375 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.761268 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a85861de-cb4f-4da6-ad76-4e55e957d797" containerName="payload-processing" Nov 04 04:01:20.761375 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.761282 4966 cpu_manager.go:414] "containerMap: removing container" podUID="6d5ce632-2597-47ef-bb2f-dd41f330c12f" containerName="payload-pre-processing" Nov 04 04:01:20.761375 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.761289 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d5ce632-2597-47ef-bb2f-dd41f330c12f" containerName="payload-pre-processing" Nov 04 04:01:20.761375 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.761374 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a85861de-cb4f-4da6-ad76-4e55e957d797" containerName="payload-processing" Nov 04 04:01:20.761558 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.761383 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d5ce632-2597-47ef-bb2f-dd41f330c12f" containerName="payload-pre-processing" Nov 04 04:01:20.761558 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.761391 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3324a6f-e5f3-4ea0-9e43-e2900ae045f1" containerName="maas-api" Nov 04 04:01:20.766170 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.766150 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.772718 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.772676 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-56a44f-openshift-default-dockercfg-7cnql\"" Nov 04 04:01:20.777539 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.777329 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8"] Nov 04 04:01:20.876671 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.876579 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-credential-socket\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.876671 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.876630 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-token\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.876882 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.876715 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-podinfo\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.876882 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.876747 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-workload-socket\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.876882 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.876771 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-workload-certs\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.876882 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.876867 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr8j8\" (UniqueName: \"kubernetes.io/projected/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-kube-api-access-nr8j8\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.877052 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.877010 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istiod-ca-cert\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.877052 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.877043 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-data\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.877127 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.877075 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-envoy\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.978121 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.978075 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istiod-ca-cert\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.978308 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.978128 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-data\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.978308 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.978161 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-envoy\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.978308 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.978239 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-credential-socket\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.978308 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.978268 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-token\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.978308 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.978293 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-podinfo\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.978556 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.978329 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-workload-socket\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.978556 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.978503 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-workload-certs\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.978653 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.978567 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr8j8\" (UniqueName: \"kubernetes.io/projected/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-kube-api-access-nr8j8\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.978853 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.978832 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istiod-ca-cert\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.979066 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.979038 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-data\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.979202 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.979086 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-workload-socket\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.979202 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.979169 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-workload-certs\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.979369 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.979345 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-credential-socket\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.980813 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.980790 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-envoy\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.981249 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.981230 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-podinfo\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.990470 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.990430 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-token\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:20.991213 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:20.991190 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr8j8\" (UniqueName: \"kubernetes.io/projected/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-kube-api-access-nr8j8\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:21.082502 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:21.082469 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:21.368864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:21.368839 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8"] Nov 04 04:01:21.370562 ip-10-0-138-9 kubenswrapper[4966]: W1104 04:01:21.370525 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1303a0_cb6b_4d05_9b03_bc3eb0e62bd4.slice/crio-fb6d9a9f3030b1b87cc3afe4208a921e456ce31fafadc35ef4d497d19cc0e928 WatchSource:0}: Error finding container fb6d9a9f3030b1b87cc3afe4208a921e456ce31fafadc35ef4d497d19cc0e928: Status 404 returned error can't find the container with id fb6d9a9f3030b1b87cc3afe4208a921e456ce31fafadc35ef4d497d19cc0e928 Nov 04 04:01:21.373183 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:21.373139 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:01:21.373302 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:21.373238 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:01:21.373302 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:21.373291 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:01:21.559181 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:21.559132 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" event={"ID":"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4","Type":"ContainerStarted","Data":"2a2401eb60d2d683a092245c56a74c2a7240c450835813bfe3eccf29ebd2848e"} Nov 04 04:01:21.559367 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:21.559200 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" event={"ID":"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4","Type":"ContainerStarted","Data":"fb6d9a9f3030b1b87cc3afe4208a921e456ce31fafadc35ef4d497d19cc0e928"} Nov 04 04:01:21.585223 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:21.585167 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" podStartSLOduration=1.5851527029999999 podStartE2EDuration="1.585152703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:01:21.583555408 +0000 UTC m=+725.384501230" watchObservedRunningTime="2026-11-04 04:01:21.585152703 +0000 UTC m=+725.386098511" Nov 04 04:01:22.083287 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:22.083245 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:22.085068 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:22.085038 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 04:01:22.085209 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:22.085095 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" podUID="8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 04:01:22.536956 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:22.536860 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms"] Nov 04 04:01:22.537532 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:22.537472 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" podUID="016f8a5d-06ed-4e0c-9a36-f541c1232337" containerName="maas-api" containerID="cri-o://afdc154ef92311df143b3988b77416cde575ea3cb12aea4d93a07c536829bac8" gracePeriod=30 Nov 04 04:01:22.686387 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:22.685605 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v"] Nov 04 04:01:22.686387 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:22.685874 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" podUID="d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2" containerName="maas-api" containerID="cri-o://8689d8433b360a49de25d939a46622d55731c4c8bae8d8a4d765293fd96edd3c" gracePeriod=30 Nov 04 04:01:22.932459 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:22.932424 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" Nov 04 04:01:23.023159 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.023110 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/016f8a5d-06ed-4e0c-9a36-f541c1232337-kube-api-access-gdbgs\" (UniqueName: \"kubernetes.io/projected/016f8a5d-06ed-4e0c-9a36-f541c1232337-kube-api-access-gdbgs\") pod \"016f8a5d-06ed-4e0c-9a36-f541c1232337\" (UID: \"016f8a5d-06ed-4e0c-9a36-f541c1232337\") " Nov 04 04:01:23.032975 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.027497 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/016f8a5d-06ed-4e0c-9a36-f541c1232337-kube-api-access-gdbgs" pod "016f8a5d-06ed-4e0c-9a36-f541c1232337" (UID: "016f8a5d-06ed-4e0c-9a36-f541c1232337"). InnerVolumeSpecName "kube-api-access-gdbgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:01:23.083778 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.083653 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 04:01:23.084270 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.083711 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" podUID="8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 04:01:23.090970 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.090945 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" Nov 04 04:01:23.123804 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.123764 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/016f8a5d-06ed-4e0c-9a36-f541c1232337-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016f8a5d-06ed-4e0c-9a36-f541c1232337-openshift-service-ca\") pod \"016f8a5d-06ed-4e0c-9a36-f541c1232337\" (UID: \"016f8a5d-06ed-4e0c-9a36-f541c1232337\") " Nov 04 04:01:23.123979 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.123892 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-openshift-service-ca\") pod \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\" (UID: \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\") " Nov 04 04:01:23.124044 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.123996 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-kube-api-access-kbss2\" (UniqueName: \"kubernetes.io/projected/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-kube-api-access-kbss2\") pod \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\" (UID: \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\") " Nov 04 04:01:23.124100 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.124054 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-maas-api-tls\") pod \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\" (UID: \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\") " Nov 04 04:01:23.124100 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.124084 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-metrics-certs\") pod \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\" (UID: \"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2\") " Nov 04 04:01:23.124323 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.124084 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/016f8a5d-06ed-4e0c-9a36-f541c1232337-openshift-service-ca" pod "016f8a5d-06ed-4e0c-9a36-f541c1232337" (UID: "016f8a5d-06ed-4e0c-9a36-f541c1232337"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:01:23.124724 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.124680 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-openshift-service-ca" pod "d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2" (UID: "d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:01:23.124724 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.124687 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/016f8a5d-06ed-4e0c-9a36-f541c1232337-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016f8a5d-06ed-4e0c-9a36-f541c1232337-maas-api-tls\") pod \"016f8a5d-06ed-4e0c-9a36-f541c1232337\" (UID: \"016f8a5d-06ed-4e0c-9a36-f541c1232337\") " Nov 04 04:01:23.124906 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.124756 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/016f8a5d-06ed-4e0c-9a36-f541c1232337-metrics-certs\" (UniqueName: \"kubernetes.io/secret/016f8a5d-06ed-4e0c-9a36-f541c1232337-metrics-certs\") pod \"016f8a5d-06ed-4e0c-9a36-f541c1232337\" (UID: \"016f8a5d-06ed-4e0c-9a36-f541c1232337\") " Nov 04 04:01:23.125256 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.125228 4966 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016f8a5d-06ed-4e0c-9a36-f541c1232337-openshift-service-ca\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:23.125335 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.125261 4966 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-openshift-service-ca\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:23.125335 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.125275 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdbgs\" (UniqueName: \"kubernetes.io/projected/016f8a5d-06ed-4e0c-9a36-f541c1232337-kube-api-access-gdbgs\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:23.127406 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.127377 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/016f8a5d-06ed-4e0c-9a36-f541c1232337-metrics-certs" pod "016f8a5d-06ed-4e0c-9a36-f541c1232337" (UID: "016f8a5d-06ed-4e0c-9a36-f541c1232337"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:01:23.127506 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.127435 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-metrics-certs" pod "d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2" (UID: "d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:01:23.127506 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.127494 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/016f8a5d-06ed-4e0c-9a36-f541c1232337-maas-api-tls" pod "016f8a5d-06ed-4e0c-9a36-f541c1232337" (UID: "016f8a5d-06ed-4e0c-9a36-f541c1232337"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:01:23.127799 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.127774 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-kube-api-access-kbss2" pod "d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2" (UID: "d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2"). InnerVolumeSpecName "kube-api-access-kbss2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:01:23.128821 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.128800 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-maas-api-tls" pod "d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2" (UID: "d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:01:23.226227 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.226192 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbss2\" (UniqueName: \"kubernetes.io/projected/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-kube-api-access-kbss2\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:23.226227 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.226228 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-maas-api-tls\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:23.226499 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.226238 4966 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2-metrics-certs\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:23.226499 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.226248 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016f8a5d-06ed-4e0c-9a36-f541c1232337-maas-api-tls\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:23.226499 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.226257 4966 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016f8a5d-06ed-4e0c-9a36-f541c1232337-metrics-certs\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:23.603800 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.603765 4966 generic.go:356] "Generic (PLEG): container finished" podID="016f8a5d-06ed-4e0c-9a36-f541c1232337" containerID="afdc154ef92311df143b3988b77416cde575ea3cb12aea4d93a07c536829bac8" exitCode=0 Nov 04 04:01:23.604006 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.603812 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" Nov 04 04:01:23.604006 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.603853 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" event={"ID":"016f8a5d-06ed-4e0c-9a36-f541c1232337","Type":"ContainerDied","Data":"afdc154ef92311df143b3988b77416cde575ea3cb12aea4d93a07c536829bac8"} Nov 04 04:01:23.604006 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.603914 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms" event={"ID":"016f8a5d-06ed-4e0c-9a36-f541c1232337","Type":"ContainerDied","Data":"1076cfe62518d06ba08847c17299d287777b2d9db26a657089ab583ce00d0ed7"} Nov 04 04:01:23.604006 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.603975 4966 scope.go:122] "RemoveContainer" containerID="afdc154ef92311df143b3988b77416cde575ea3cb12aea4d93a07c536829bac8" Nov 04 04:01:23.605535 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.605403 4966 generic.go:356] "Generic (PLEG): container finished" podID="d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2" containerID="8689d8433b360a49de25d939a46622d55731c4c8bae8d8a4d765293fd96edd3c" exitCode=0 Nov 04 04:01:23.605535 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.605444 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" Nov 04 04:01:23.605535 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.605478 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" event={"ID":"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2","Type":"ContainerDied","Data":"8689d8433b360a49de25d939a46622d55731c4c8bae8d8a4d765293fd96edd3c"} Nov 04 04:01:23.605535 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.605520 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v" event={"ID":"d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2","Type":"ContainerDied","Data":"40ccd6b1270da18b3ea9523ce0a75806b48dc316652a02005d8d304df0e08bce"} Nov 04 04:01:23.617156 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.617138 4966 scope.go:122] "RemoveContainer" containerID="afdc154ef92311df143b3988b77416cde575ea3cb12aea4d93a07c536829bac8" Nov 04 04:01:23.617496 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:01:23.617472 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afdc154ef92311df143b3988b77416cde575ea3cb12aea4d93a07c536829bac8\": container with ID starting with afdc154ef92311df143b3988b77416cde575ea3cb12aea4d93a07c536829bac8 not found: ID does not exist" containerID="afdc154ef92311df143b3988b77416cde575ea3cb12aea4d93a07c536829bac8" Nov 04 04:01:23.617587 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.617507 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afdc154ef92311df143b3988b77416cde575ea3cb12aea4d93a07c536829bac8"} err="failed to get container status \"afdc154ef92311df143b3988b77416cde575ea3cb12aea4d93a07c536829bac8\": rpc error: code = NotFound desc = could not find container \"afdc154ef92311df143b3988b77416cde575ea3cb12aea4d93a07c536829bac8\": container with ID starting with afdc154ef92311df143b3988b77416cde575ea3cb12aea4d93a07c536829bac8 not found: ID does not exist" Nov 04 04:01:23.617587 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.617527 4966 scope.go:122] "RemoveContainer" containerID="8689d8433b360a49de25d939a46622d55731c4c8bae8d8a4d765293fd96edd3c" Nov 04 04:01:23.626980 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.626957 4966 scope.go:122] "RemoveContainer" containerID="8689d8433b360a49de25d939a46622d55731c4c8bae8d8a4d765293fd96edd3c" Nov 04 04:01:23.630826 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:01:23.630795 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8689d8433b360a49de25d939a46622d55731c4c8bae8d8a4d765293fd96edd3c\": container with ID starting with 8689d8433b360a49de25d939a46622d55731c4c8bae8d8a4d765293fd96edd3c not found: ID does not exist" containerID="8689d8433b360a49de25d939a46622d55731c4c8bae8d8a4d765293fd96edd3c" Nov 04 04:01:23.630981 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.630834 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8689d8433b360a49de25d939a46622d55731c4c8bae8d8a4d765293fd96edd3c"} err="failed to get container status \"8689d8433b360a49de25d939a46622d55731c4c8bae8d8a4d765293fd96edd3c\": rpc error: code = NotFound desc = could not find container \"8689d8433b360a49de25d939a46622d55731c4c8bae8d8a4d765293fd96edd3c\": container with ID starting with 8689d8433b360a49de25d939a46622d55731c4c8bae8d8a4d765293fd96edd3c not found: ID does not exist" Nov 04 04:01:23.634361 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.634339 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms"] Nov 04 04:01:23.636872 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.636853 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-78ca37-6dcb7fb968-km5ms"] Nov 04 04:01:23.648851 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.648679 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v"] Nov 04 04:01:23.651971 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:23.651947 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-36730f-6ffdbcff9-5kk5v"] Nov 04 04:01:24.085381 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:24.085344 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 04:01:24.086003 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:24.085972 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" podUID="8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 04:01:24.342326 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:24.342245 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="016f8a5d-06ed-4e0c-9a36-f541c1232337" path="/var/lib/kubelet/pods/016f8a5d-06ed-4e0c-9a36-f541c1232337/volumes" Nov 04 04:01:24.342702 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:24.342687 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2" path="/var/lib/kubelet/pods/d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2/volumes" Nov 04 04:01:25.084121 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:25.083605 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 04:01:25.084121 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:25.083684 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" podUID="8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 04:01:26.083956 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:26.083449 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 04:01:26.083956 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:26.083524 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" podUID="8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 04:01:26.094953 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:26.093152 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw"] Nov 04 04:01:27.086126 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:27.086079 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 04:01:27.086740 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:27.086705 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" podUID="8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 04:01:28.083490 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:28.083457 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 04:01:28.083663 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:28.083513 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" podUID="8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 04:01:28.421352 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:28.421256 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" podUID="e8517352-f4dc-4e65-8ac7-ccb0b01d5255" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 04:01:29.392553 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:29.392515 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:29.392986 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:29.392954 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:29.393754 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:29.393732 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:01:29.421261 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:29.421232 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7"] Nov 04 04:01:29.448786 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:29.448750 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh"] Nov 04 04:01:31.978296 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:31.978258 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp"] Nov 04 04:01:32.436581 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:32.436543 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" Nov 04 04:01:32.589036 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:32.588172 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8517352-f4dc-4e65-8ac7-ccb0b01d5255-kube-api-access-dsmzz\" (UniqueName: \"kubernetes.io/projected/e8517352-f4dc-4e65-8ac7-ccb0b01d5255-kube-api-access-dsmzz\") pod \"e8517352-f4dc-4e65-8ac7-ccb0b01d5255\" (UID: \"e8517352-f4dc-4e65-8ac7-ccb0b01d5255\") " Nov 04 04:01:32.589036 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:32.588264 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8517352-f4dc-4e65-8ac7-ccb0b01d5255-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8517352-f4dc-4e65-8ac7-ccb0b01d5255-plugins-config-volume\") pod \"e8517352-f4dc-4e65-8ac7-ccb0b01d5255\" (UID: \"e8517352-f4dc-4e65-8ac7-ccb0b01d5255\") " Nov 04 04:01:32.589595 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:32.589569 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8517352-f4dc-4e65-8ac7-ccb0b01d5255-plugins-config-volume" pod "e8517352-f4dc-4e65-8ac7-ccb0b01d5255" (UID: "e8517352-f4dc-4e65-8ac7-ccb0b01d5255"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:01:32.592565 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:32.592538 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8517352-f4dc-4e65-8ac7-ccb0b01d5255-kube-api-access-dsmzz" pod "e8517352-f4dc-4e65-8ac7-ccb0b01d5255" (UID: "e8517352-f4dc-4e65-8ac7-ccb0b01d5255"). InnerVolumeSpecName "kube-api-access-dsmzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:01:32.690777 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:32.690738 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsmzz\" (UniqueName: \"kubernetes.io/projected/e8517352-f4dc-4e65-8ac7-ccb0b01d5255-kube-api-access-dsmzz\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:32.690777 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:32.690779 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8517352-f4dc-4e65-8ac7-ccb0b01d5255-plugins-config-volume\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:32.706824 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:32.706786 4966 generic.go:356] "Generic (PLEG): container finished" podID="e8517352-f4dc-4e65-8ac7-ccb0b01d5255" containerID="398b12ee0e5eceb08f69a5e4482c133a7a718d9dce147060980cf364bf6b2228" exitCode=0 Nov 04 04:01:32.707061 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:32.706854 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" Nov 04 04:01:32.707061 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:32.706862 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" event={"ID":"e8517352-f4dc-4e65-8ac7-ccb0b01d5255","Type":"ContainerDied","Data":"398b12ee0e5eceb08f69a5e4482c133a7a718d9dce147060980cf364bf6b2228"} Nov 04 04:01:32.707061 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:32.706955 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg" event={"ID":"e8517352-f4dc-4e65-8ac7-ccb0b01d5255","Type":"ContainerDied","Data":"7ce95ea356082b83444524cc4d03a0fadf88cd1803c99f3b729190c821aaa693"} Nov 04 04:01:32.707061 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:32.706987 4966 scope.go:122] "RemoveContainer" containerID="398b12ee0e5eceb08f69a5e4482c133a7a718d9dce147060980cf364bf6b2228" Nov 04 04:01:32.719614 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:32.719591 4966 scope.go:122] "RemoveContainer" containerID="398b12ee0e5eceb08f69a5e4482c133a7a718d9dce147060980cf364bf6b2228" Nov 04 04:01:32.719940 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:01:32.719905 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"398b12ee0e5eceb08f69a5e4482c133a7a718d9dce147060980cf364bf6b2228\": container with ID starting with 398b12ee0e5eceb08f69a5e4482c133a7a718d9dce147060980cf364bf6b2228 not found: ID does not exist" containerID="398b12ee0e5eceb08f69a5e4482c133a7a718d9dce147060980cf364bf6b2228" Nov 04 04:01:32.720033 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:32.719950 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"398b12ee0e5eceb08f69a5e4482c133a7a718d9dce147060980cf364bf6b2228"} err="failed to get container status \"398b12ee0e5eceb08f69a5e4482c133a7a718d9dce147060980cf364bf6b2228\": rpc error: code = NotFound desc = could not find container \"398b12ee0e5eceb08f69a5e4482c133a7a718d9dce147060980cf364bf6b2228\": container with ID starting with 398b12ee0e5eceb08f69a5e4482c133a7a718d9dce147060980cf364bf6b2228 not found: ID does not exist" Nov 04 04:01:32.734478 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:32.734411 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg"] Nov 04 04:01:32.738903 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:32.738857 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg"] Nov 04 04:01:34.341762 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:34.341724 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8517352-f4dc-4e65-8ac7-ccb0b01d5255" path="/var/lib/kubelet/pods/e8517352-f4dc-4e65-8ac7-ccb0b01d5255/volumes" Nov 04 04:01:35.600217 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.600182 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv"] Nov 04 04:01:35.601047 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.601027 4966 cpu_manager.go:414] "containerMap: removing container" podUID="d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2" containerName="maas-api" Nov 04 04:01:35.601114 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.601050 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2" containerName="maas-api" Nov 04 04:01:35.601114 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.601086 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e8517352-f4dc-4e65-8ac7-ccb0b01d5255" containerName="payload-processing" Nov 04 04:01:35.601114 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.601094 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8517352-f4dc-4e65-8ac7-ccb0b01d5255" containerName="payload-processing" Nov 04 04:01:35.601205 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.601152 4966 cpu_manager.go:414] "containerMap: removing container" podUID="016f8a5d-06ed-4e0c-9a36-f541c1232337" containerName="maas-api" Nov 04 04:01:35.601205 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.601160 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="016f8a5d-06ed-4e0c-9a36-f541c1232337" containerName="maas-api" Nov 04 04:01:35.601301 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.601289 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8517352-f4dc-4e65-8ac7-ccb0b01d5255" containerName="payload-processing" Nov 04 04:01:35.601333 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.601313 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="016f8a5d-06ed-4e0c-9a36-f541c1232337" containerName="maas-api" Nov 04 04:01:35.601333 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.601323 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d910f8fa-7ab2-44c5-b175-a56bf0e0d6b2" containerName="maas-api" Nov 04 04:01:35.607149 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.607116 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.623781 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.623753 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv"] Nov 04 04:01:35.663843 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.663809 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd"] Nov 04 04:01:35.668650 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.668626 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.681430 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.681402 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-4afe0e-openshift-default-dockercfg-x52mt\"" Nov 04 04:01:35.684527 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:01:35.684486 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 04:01:35.686096 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.686073 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd"] Nov 04 04:01:35.733769 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.733727 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-workload-certs\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.733969 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.733790 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388ae503-75e8-4507-b6c2-9e4229c67128-istio-token\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.733969 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.733863 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-credential-socket\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.733969 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.733901 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388ae503-75e8-4507-b6c2-9e4229c67128-istiod-ca-cert\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.733969 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.733938 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388ae503-75e8-4507-b6c2-9e4229c67128-istio-podinfo\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.734099 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.734046 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-workload-socket\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.734099 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.734086 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44l7q\" (UniqueName: \"kubernetes.io/projected/388ae503-75e8-4507-b6c2-9e4229c67128-kube-api-access-44l7q\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.734197 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.734184 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-istio-envoy\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.734232 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.734210 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-istio-data\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.806780 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.806745 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn"] Nov 04 04:01:35.811205 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.811175 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:35.835109 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.835077 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn"] Nov 04 04:01:35.835557 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.835530 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-workload-socket\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.835602 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.835587 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-workload-certs\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.835658 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.835644 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388ae503-75e8-4507-b6c2-9e4229c67128-istio-token\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.835691 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.835674 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-credential-socket\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.835724 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.835705 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388ae503-75e8-4507-b6c2-9e4229c67128-istiod-ca-cert\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.835763 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.835739 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388ae503-75e8-4507-b6c2-9e4229c67128-istio-podinfo\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.835819 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.835769 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-token\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.835819 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.835807 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-credential-socket\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.835901 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.835835 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-envoy\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.835901 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.835873 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-podinfo\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.836033 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.835902 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-workload-socket\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.836033 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.835972 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44l7q\" (UniqueName: \"kubernetes.io/projected/388ae503-75e8-4507-b6c2-9e4229c67128-kube-api-access-44l7q\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.836033 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.836005 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-data\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.836170 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.836038 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h7rl\" (UniqueName: \"kubernetes.io/projected/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-kube-api-access-6h7rl\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.836170 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.836096 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-workload-certs\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.836170 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.836135 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istiod-ca-cert\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.836319 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.836180 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-istio-envoy\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.836319 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.836220 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-istio-data\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.836651 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.836629 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-istio-data\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.836842 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.836822 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-workload-certs\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.836933 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.836882 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-workload-socket\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.837528 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.837432 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-credential-socket\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.838129 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.838097 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388ae503-75e8-4507-b6c2-9e4229c67128-istiod-ca-cert\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.839905 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.839867 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-istio-envoy\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.842503 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.842474 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388ae503-75e8-4507-b6c2-9e4229c67128-istio-podinfo\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.852909 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.852851 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44l7q\" (UniqueName: \"kubernetes.io/projected/388ae503-75e8-4507-b6c2-9e4229c67128-kube-api-access-44l7q\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.853242 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.853220 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388ae503-75e8-4507-b6c2-9e4229c67128-istio-token\") pod \"e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.934606 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.934570 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:35.937282 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.937244 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-workload-socket\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.937463 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.937372 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-workload-socket\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:35.937463 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.937403 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-credential-socket\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:35.937463 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.937434 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2060984f-5a85-4790-b19c-3e6fff9016ae-istiod-ca-cert\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:35.937628 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.937606 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-token\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.937704 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.937648 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-token\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:35.937784 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.937701 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-credential-socket\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.937784 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.937726 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-workload-certs\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:35.937784 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.937751 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-envoy\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.937784 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.937778 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-podinfo\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:35.938024 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.937806 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-data\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:35.938024 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.937836 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-podinfo\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.938024 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.937902 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-data\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.938024 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.937970 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h7rl\" (UniqueName: \"kubernetes.io/projected/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-kube-api-access-6h7rl\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.938180 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.938040 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-envoy\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:35.938180 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.938093 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-workload-certs\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.938180 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.938126 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl27c\" (UniqueName: \"kubernetes.io/projected/2060984f-5a85-4790-b19c-3e6fff9016ae-kube-api-access-gl27c\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:35.938180 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.938172 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istiod-ca-cert\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.938379 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.938354 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-workload-socket\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.941570 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.939066 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istiod-ca-cert\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.941570 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.939304 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-data\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.941570 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.939589 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-credential-socket\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.941570 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.939791 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-workload-certs\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.942298 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.942269 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-podinfo\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.946190 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.946163 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-envoy\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.950368 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.950333 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h7rl\" (UniqueName: \"kubernetes.io/projected/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-kube-api-access-6h7rl\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.950577 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.950555 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-token\") pod \"e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:35.988852 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:35.988815 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:36.039850 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.039813 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-workload-socket\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:36.040063 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.039863 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-credential-socket\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:36.040063 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.039888 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2060984f-5a85-4790-b19c-3e6fff9016ae-istiod-ca-cert\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:36.040063 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.039907 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-token\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:36.040063 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.039948 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-workload-certs\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:36.040063 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.039971 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-podinfo\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:36.040063 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.039991 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-data\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:36.040063 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.040043 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-envoy\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:36.040318 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.040070 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl27c\" (UniqueName: \"kubernetes.io/projected/2060984f-5a85-4790-b19c-3e6fff9016ae-kube-api-access-gl27c\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:36.040767 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.040743 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-workload-socket\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:36.041295 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.041264 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-workload-certs\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:36.041389 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.041298 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-credential-socket\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:36.041733 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.041712 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-data\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:36.042158 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.042133 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2060984f-5a85-4790-b19c-3e6fff9016ae-istiod-ca-cert\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:36.045393 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.045185 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-envoy\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:36.045393 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.045363 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-podinfo\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:36.056192 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.056165 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl27c\" (UniqueName: \"kubernetes.io/projected/2060984f-5a85-4790-b19c-3e6fff9016ae-kube-api-access-gl27c\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:36.056725 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.056701 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-token\") pod \"e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:36.133865 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.133774 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:36.236447 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.236412 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv"] Nov 04 04:01:36.237590 ip-10-0-138-9 kubenswrapper[4966]: W1104 04:01:36.237560 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388ae503_75e8_4507_b6c2_9e4229c67128.slice/crio-e9d6c8f9f27dbf389264bc096115adc61edda2c4217eaac89f109a8698cdbea6 WatchSource:0}: Error finding container e9d6c8f9f27dbf389264bc096115adc61edda2c4217eaac89f109a8698cdbea6: Status 404 returned error can't find the container with id e9d6c8f9f27dbf389264bc096115adc61edda2c4217eaac89f109a8698cdbea6 Nov 04 04:01:36.240601 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.240563 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:01:36.240714 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.240667 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:01:36.240803 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.240749 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:01:36.296860 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.296831 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd"] Nov 04 04:01:36.300299 ip-10-0-138-9 kubenswrapper[4966]: W1104 04:01:36.300272 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0e1ba5_30ca_49f3_be4b_2e5a474d036a.slice/crio-dbe1b2e9823a1f7a82686dce8ba23c9926fc1c3e2c15dd87a9cd3f073cbed410 WatchSource:0}: Error finding container dbe1b2e9823a1f7a82686dce8ba23c9926fc1c3e2c15dd87a9cd3f073cbed410: Status 404 returned error can't find the container with id dbe1b2e9823a1f7a82686dce8ba23c9926fc1c3e2c15dd87a9cd3f073cbed410 Nov 04 04:01:36.303008 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.302977 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:01:36.303160 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.303048 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:01:36.303160 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.303081 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:01:36.425885 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.425536 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn"] Nov 04 04:01:36.428143 ip-10-0-138-9 kubenswrapper[4966]: W1104 04:01:36.428113 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2060984f_5a85_4790_b19c_3e6fff9016ae.slice/crio-749904aafe752d4f7be5af6c402098a99a9a20e45ab67f7c0b06cbb597bf6ca5 WatchSource:0}: Error finding container 749904aafe752d4f7be5af6c402098a99a9a20e45ab67f7c0b06cbb597bf6ca5: Status 404 returned error can't find the container with id 749904aafe752d4f7be5af6c402098a99a9a20e45ab67f7c0b06cbb597bf6ca5 Nov 04 04:01:36.432091 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.431583 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:01:36.432091 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.431669 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:01:36.432091 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.431708 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:01:36.470091 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.470059 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8"] Nov 04 04:01:36.746724 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.746621 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" event={"ID":"2060984f-5a85-4790-b19c-3e6fff9016ae","Type":"ContainerStarted","Data":"8cbf6ce60026f824f759368dba5101dc9d8a3391668343b29bc4b96600760d17"} Nov 04 04:01:36.747230 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.746733 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" event={"ID":"2060984f-5a85-4790-b19c-3e6fff9016ae","Type":"ContainerStarted","Data":"749904aafe752d4f7be5af6c402098a99a9a20e45ab67f7c0b06cbb597bf6ca5"} Nov 04 04:01:36.760665 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.760621 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" event={"ID":"388ae503-75e8-4507-b6c2-9e4229c67128","Type":"ContainerStarted","Data":"629ecbdb3ffcb8bbbbc4b67f44531a847a4c37dd509310da4d1f2b752a547841"} Nov 04 04:01:36.761008 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.760974 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" event={"ID":"388ae503-75e8-4507-b6c2-9e4229c67128","Type":"ContainerStarted","Data":"e9d6c8f9f27dbf389264bc096115adc61edda2c4217eaac89f109a8698cdbea6"} Nov 04 04:01:36.764127 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.764097 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" event={"ID":"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a","Type":"ContainerStarted","Data":"1a84f817ae0b87d071668a645713431f560e0c76d3367f81bf18240a467e77a8"} Nov 04 04:01:36.765102 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.765077 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" event={"ID":"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a","Type":"ContainerStarted","Data":"dbe1b2e9823a1f7a82686dce8ba23c9926fc1c3e2c15dd87a9cd3f073cbed410"} Nov 04 04:01:36.791242 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.791171 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" podStartSLOduration=1.791149249 podStartE2EDuration="1.791149249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:01:36.788863964 +0000 UTC m=+740.589809773" watchObservedRunningTime="2026-11-04 04:01:36.791149249 +0000 UTC m=+740.592095059" Nov 04 04:01:36.819840 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.819771 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" podStartSLOduration=1.819747869 podStartE2EDuration="1.819747869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:01:36.817020221 +0000 UTC m=+740.617966029" watchObservedRunningTime="2026-11-04 04:01:36.819747869 +0000 UTC m=+740.620693678" Nov 04 04:01:36.936117 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.936074 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:36.984805 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.984757 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" podUID="28e01b06-4f41-4ec9-b351-6145f5d0bd2a" containerName="payload-pre-processing" containerID="cri-o://641b44eef565e6d3d1eccc4b26631e2eef89384deddacce49e0b2c14471ab5ac" gracePeriod=25 Nov 04 04:01:36.995173 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:36.995140 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:37.140053 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:37.140020 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:37.145424 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:37.145392 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 04:01:37.145654 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:37.145628 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" podUID="2060984f-5a85-4790-b19c-3e6fff9016ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 04:01:37.940882 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:37.940844 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 04 04:01:37.950071 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:37.950027 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" podUID="388ae503-75e8-4507-b6c2-9e4229c67128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 04 04:01:38.013220 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:38.013185 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 04 04:01:38.013490 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:38.013463 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" podUID="1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 04 04:01:38.204005 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:38.195957 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" podStartSLOduration=3.19593385 podStartE2EDuration="3.19593385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:01:36.848612697 +0000 UTC m=+740.649558506" watchObservedRunningTime="2026-11-04 04:01:38.19593385 +0000 UTC m=+741.996879649" Nov 04 04:01:38.387944 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:38.382601 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 04:01:38.387944 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:38.382666 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" podUID="2060984f-5a85-4790-b19c-3e6fff9016ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 04:01:38.506001 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:38.504473 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl"] Nov 04 04:01:38.506001 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:38.504721 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" podUID="a7f1c31b-8f96-4c56-ab05-8e22896d6533" containerName="istio-proxy" containerID="cri-o://690639b0b4db94b94d5d56158a9fcd34b4c7c8f6f34d792764c4bc985dc0ae29" gracePeriod=30 Nov 04 04:01:38.950422 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:38.950389 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 04 04:01:38.951019 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:38.950988 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" podUID="388ae503-75e8-4507-b6c2-9e4229c67128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 04 04:01:39.006799 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:39.006676 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 04 04:01:39.007591 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:39.007069 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" podUID="1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 04 04:01:39.138943 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:39.136472 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 04:01:39.138943 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:39.136552 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" podUID="2060984f-5a85-4790-b19c-3e6fff9016ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 04:01:39.940516 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:39.940464 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 04 04:01:39.940817 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:39.940777 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" podUID="388ae503-75e8-4507-b6c2-9e4229c67128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 04 04:01:40.008953 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:39.997635 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 04 04:01:40.009561 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:40.009521 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" podUID="1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 04 04:01:40.137600 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:40.137564 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 04:01:40.137892 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:40.137857 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" podUID="2060984f-5a85-4790-b19c-3e6fff9016ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 04:01:40.938747 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:40.937550 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 04 04:01:40.938747 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:40.937602 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" podUID="388ae503-75e8-4507-b6c2-9e4229c67128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 04 04:01:41.001391 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:41.001352 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 04 04:01:41.001694 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:41.001663 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" podUID="1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 04 04:01:41.135349 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:41.135308 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 04:01:41.135865 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:41.135374 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" podUID="2060984f-5a85-4790-b19c-3e6fff9016ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 04:01:41.151150 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:41.150488 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" podUID="eed1e3d9-e514-4d8f-b1d3-851453900699" containerName="main" containerID="cri-o://ae9c302fde91d5bede7b0be5e4d5042de4338188dbabff451a02fd53f4e578f2" gracePeriod=45 Nov 04 04:01:41.290103 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:41.288634 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" podUID="eed1e3d9-e514-4d8f-b1d3-851453900699" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": dial tcp 10.132.0.76:8000: connect: connection refused" Nov 04 04:01:41.942937 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:41.939058 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 04 04:01:41.942937 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:41.939119 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" podUID="388ae503-75e8-4507-b6c2-9e4229c67128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 04 04:01:41.952799 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:41.952148 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:01:42.037802 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.037761 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 04 04:01:42.038373 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.038340 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" podUID="1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 04 04:01:42.038618 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.038587 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-dshm\") pod \"eed1e3d9-e514-4d8f-b1d3-851453900699\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " Nov 04 04:01:42.038711 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.038643 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-tmp-dir\") pod \"eed1e3d9-e514-4d8f-b1d3-851453900699\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " Nov 04 04:01:42.038769 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.038752 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-model-cache\") pod \"eed1e3d9-e514-4d8f-b1d3-851453900699\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " Nov 04 04:01:42.038822 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.038794 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eed1e3d9-e514-4d8f-b1d3-851453900699-tls-certs\" (UniqueName: \"kubernetes.io/secret/eed1e3d9-e514-4d8f-b1d3-851453900699-tls-certs\") pod \"eed1e3d9-e514-4d8f-b1d3-851453900699\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " Nov 04 04:01:42.038873 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.038826 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-home\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-home\") pod \"eed1e3d9-e514-4d8f-b1d3-851453900699\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " Nov 04 04:01:42.038946 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.038886 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed1e3d9-e514-4d8f-b1d3-851453900699-kube-api-access-pxh9b\" (UniqueName: \"kubernetes.io/projected/eed1e3d9-e514-4d8f-b1d3-851453900699-kube-api-access-pxh9b\") pod \"eed1e3d9-e514-4d8f-b1d3-851453900699\" (UID: \"eed1e3d9-e514-4d8f-b1d3-851453900699\") " Nov 04 04:01:42.041362 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.041300 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-home" pod "eed1e3d9-e514-4d8f-b1d3-851453900699" (UID: "eed1e3d9-e514-4d8f-b1d3-851453900699"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:42.041712 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.041691 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-tmp-dir" pod "eed1e3d9-e514-4d8f-b1d3-851453900699" (UID: "eed1e3d9-e514-4d8f-b1d3-851453900699"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:42.047140 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.047096 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-model-cache" pod "eed1e3d9-e514-4d8f-b1d3-851453900699" (UID: "eed1e3d9-e514-4d8f-b1d3-851453900699"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:42.047664 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.047616 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed1e3d9-e514-4d8f-b1d3-851453900699-tls-certs" pod "eed1e3d9-e514-4d8f-b1d3-851453900699" (UID: "eed1e3d9-e514-4d8f-b1d3-851453900699"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:01:42.047849 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.047827 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed1e3d9-e514-4d8f-b1d3-851453900699-kube-api-access-pxh9b" pod "eed1e3d9-e514-4d8f-b1d3-851453900699" (UID: "eed1e3d9-e514-4d8f-b1d3-851453900699"). InnerVolumeSpecName "kube-api-access-pxh9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:01:42.047998 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.047984 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-dshm" pod "eed1e3d9-e514-4d8f-b1d3-851453900699" (UID: "eed1e3d9-e514-4d8f-b1d3-851453900699"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:42.138964 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.138543 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 04:01:42.138964 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.138595 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" podUID="2060984f-5a85-4790-b19c-3e6fff9016ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 04:01:42.141194 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.141108 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-tmp-dir\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:42.141194 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.141132 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-model-cache\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:42.141194 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.141148 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed1e3d9-e514-4d8f-b1d3-851453900699-tls-certs\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:42.141194 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.141159 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-home\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:42.141194 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.141167 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxh9b\" (UniqueName: \"kubernetes.io/projected/eed1e3d9-e514-4d8f-b1d3-851453900699-kube-api-access-pxh9b\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:42.141194 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.141175 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed1e3d9-e514-4d8f-b1d3-851453900699-dshm\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:42.242768 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.242728 4966 generic.go:356] "Generic (PLEG): container finished" podID="eed1e3d9-e514-4d8f-b1d3-851453900699" containerID="ae9c302fde91d5bede7b0be5e4d5042de4338188dbabff451a02fd53f4e578f2" exitCode=0 Nov 04 04:01:42.243001 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.242976 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" event={"ID":"eed1e3d9-e514-4d8f-b1d3-851453900699","Type":"ContainerDied","Data":"ae9c302fde91d5bede7b0be5e4d5042de4338188dbabff451a02fd53f4e578f2"} Nov 04 04:01:42.243204 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.243182 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" event={"ID":"eed1e3d9-e514-4d8f-b1d3-851453900699","Type":"ContainerDied","Data":"b546b480d6d1b4fa7424f5044833c509d7f47e127c2cc3d904804c262b5e4b62"} Nov 04 04:01:42.243294 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.243218 4966 scope.go:122] "RemoveContainer" containerID="ae9c302fde91d5bede7b0be5e4d5042de4338188dbabff451a02fd53f4e578f2" Nov 04 04:01:42.243355 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.243038 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw" Nov 04 04:01:42.279945 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.276040 4966 scope.go:122] "RemoveContainer" containerID="ae9c302fde91d5bede7b0be5e4d5042de4338188dbabff451a02fd53f4e578f2" Nov 04 04:01:42.301995 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:01:42.301278 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9c302fde91d5bede7b0be5e4d5042de4338188dbabff451a02fd53f4e578f2\": container with ID starting with ae9c302fde91d5bede7b0be5e4d5042de4338188dbabff451a02fd53f4e578f2 not found: ID does not exist" containerID="ae9c302fde91d5bede7b0be5e4d5042de4338188dbabff451a02fd53f4e578f2" Nov 04 04:01:42.301995 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.301334 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9c302fde91d5bede7b0be5e4d5042de4338188dbabff451a02fd53f4e578f2"} err="failed to get container status \"ae9c302fde91d5bede7b0be5e4d5042de4338188dbabff451a02fd53f4e578f2\": rpc error: code = NotFound desc = could not find container \"ae9c302fde91d5bede7b0be5e4d5042de4338188dbabff451a02fd53f4e578f2\": container with ID starting with ae9c302fde91d5bede7b0be5e4d5042de4338188dbabff451a02fd53f4e578f2 not found: ID does not exist" Nov 04 04:01:42.345729 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.345697 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw"] Nov 04 04:01:42.436859 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.436689 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-56a44f/distinct-w5-56a44f-kserve-6d69786df6-4n9dw"] Nov 04 04:01:42.935498 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.935452 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 04 04:01:42.935691 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.935527 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" podUID="388ae503-75e8-4507-b6c2-9e4229c67128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 04 04:01:42.989812 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.989777 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 04 04:01:42.990068 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:42.989838 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" podUID="1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 04 04:01:43.134842 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.134807 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 04:01:43.135316 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.135276 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" podUID="2060984f-5a85-4790-b19c-3e6fff9016ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 04:01:43.635887 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.635840 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 04 04:01:43.636438 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.635907 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" podUID="a7f1c31b-8f96-4c56-ab05-8e22896d6533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 04 04:01:43.825374 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.825057 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:01:43.876349 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.876314 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-data\") pod \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " Nov 04 04:01:43.877388 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.877364 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-token\" (UniqueName: \"kubernetes.io/projected/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-token\") pod \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " Nov 04 04:01:43.877590 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.877572 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-workload-certs\") pod \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " Nov 04 04:01:43.877733 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.877716 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-workload-socket\") pod \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " Nov 04 04:01:43.877881 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.877865 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-envoy\") pod \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " Nov 04 04:01:43.878165 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.878145 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istiod-ca-cert\") pod \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " Nov 04 04:01:43.878379 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.878359 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f1c31b-8f96-4c56-ab05-8e22896d6533-kube-api-access-zdx8l\" (UniqueName: \"kubernetes.io/projected/a7f1c31b-8f96-4c56-ab05-8e22896d6533-kube-api-access-zdx8l\") pod \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " Nov 04 04:01:43.878543 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.878526 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-podinfo\") pod \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " Nov 04 04:01:43.878730 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.878713 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-credential-socket\") pod \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\" (UID: \"a7f1c31b-8f96-4c56-ab05-8e22896d6533\") " Nov 04 04:01:43.879557 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.877275 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-data" pod "a7f1c31b-8f96-4c56-ab05-8e22896d6533" (UID: "a7f1c31b-8f96-4c56-ab05-8e22896d6533"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:43.879671 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.879532 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-credential-socket" pod "a7f1c31b-8f96-4c56-ab05-8e22896d6533" (UID: "a7f1c31b-8f96-4c56-ab05-8e22896d6533"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:43.882136 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.881849 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-workload-socket" pod "a7f1c31b-8f96-4c56-ab05-8e22896d6533" (UID: "a7f1c31b-8f96-4c56-ab05-8e22896d6533"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:43.882136 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.881949 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-workload-certs" pod "a7f1c31b-8f96-4c56-ab05-8e22896d6533" (UID: "a7f1c31b-8f96-4c56-ab05-8e22896d6533"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:43.882136 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.882099 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istiod-ca-cert" pod "a7f1c31b-8f96-4c56-ab05-8e22896d6533" (UID: "a7f1c31b-8f96-4c56-ab05-8e22896d6533"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:01:43.921505 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.921435 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-podinfo" pod "a7f1c31b-8f96-4c56-ab05-8e22896d6533" (UID: "a7f1c31b-8f96-4c56-ab05-8e22896d6533"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 04:01:43.926626 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.924393 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-envoy" pod "a7f1c31b-8f96-4c56-ab05-8e22896d6533" (UID: "a7f1c31b-8f96-4c56-ab05-8e22896d6533"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:43.926626 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.924545 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-token" pod "a7f1c31b-8f96-4c56-ab05-8e22896d6533" (UID: "a7f1c31b-8f96-4c56-ab05-8e22896d6533"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:01:43.926626 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.926554 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f1c31b-8f96-4c56-ab05-8e22896d6533-kube-api-access-zdx8l" pod "a7f1c31b-8f96-4c56-ab05-8e22896d6533" (UID: "a7f1c31b-8f96-4c56-ab05-8e22896d6533"). InnerVolumeSpecName "kube-api-access-zdx8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:01:43.980420 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.980380 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-envoy\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:43.980420 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.980423 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istiod-ca-cert\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:43.980642 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.980440 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdx8l\" (UniqueName: \"kubernetes.io/projected/a7f1c31b-8f96-4c56-ab05-8e22896d6533-kube-api-access-zdx8l\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:43.980642 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.980459 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-podinfo\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:43.980642 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.980474 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-credential-socket\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:43.980642 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.980488 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-data\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:43.980642 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.980503 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f1c31b-8f96-4c56-ab05-8e22896d6533-istio-token\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:43.980642 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.980517 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-workload-certs\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:43.980642 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:43.980533 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f1c31b-8f96-4c56-ab05-8e22896d6533-workload-socket\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:44.124944 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.124380 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf"] Nov 04 04:01:44.124944 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.124645 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" podUID="b9ae98de-5794-48e0-ae47-d93b38c29ab0" containerName="maas-api" containerID="cri-o://c097693692277ef812df5dfec02dd807044547f22510556c39546bc51d62fe4d" gracePeriod=30 Nov 04 04:01:44.266686 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.266648 4966 generic.go:356] "Generic (PLEG): container finished" podID="a7f1c31b-8f96-4c56-ab05-8e22896d6533" containerID="690639b0b4db94b94d5d56158a9fcd34b4c7c8f6f34d792764c4bc985dc0ae29" exitCode=0 Nov 04 04:01:44.266996 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.266969 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" event={"ID":"a7f1c31b-8f96-4c56-ab05-8e22896d6533","Type":"ContainerDied","Data":"690639b0b4db94b94d5d56158a9fcd34b4c7c8f6f34d792764c4bc985dc0ae29"} Nov 04 04:01:44.267187 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.267169 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" event={"ID":"a7f1c31b-8f96-4c56-ab05-8e22896d6533","Type":"ContainerDied","Data":"9c18db7b4d61fff7cea222a7c9ca4eb34898c64157a627afa3c118d016156a6d"} Nov 04 04:01:44.267322 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.267306 4966 scope.go:122] "RemoveContainer" containerID="690639b0b4db94b94d5d56158a9fcd34b4c7c8f6f34d792764c4bc985dc0ae29" Nov 04 04:01:44.267551 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.267534 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl" Nov 04 04:01:44.326887 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.326136 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl"] Nov 04 04:01:44.326887 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.326627 4966 scope.go:122] "RemoveContainer" containerID="690639b0b4db94b94d5d56158a9fcd34b4c7c8f6f34d792764c4bc985dc0ae29" Nov 04 04:01:44.332527 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:01:44.332459 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690639b0b4db94b94d5d56158a9fcd34b4c7c8f6f34d792764c4bc985dc0ae29\": container with ID starting with 690639b0b4db94b94d5d56158a9fcd34b4c7c8f6f34d792764c4bc985dc0ae29 not found: ID does not exist" containerID="690639b0b4db94b94d5d56158a9fcd34b4c7c8f6f34d792764c4bc985dc0ae29" Nov 04 04:01:44.332682 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.332511 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690639b0b4db94b94d5d56158a9fcd34b4c7c8f6f34d792764c4bc985dc0ae29"} err="failed to get container status \"690639b0b4db94b94d5d56158a9fcd34b4c7c8f6f34d792764c4bc985dc0ae29\": rpc error: code = NotFound desc = could not find container \"690639b0b4db94b94d5d56158a9fcd34b4c7c8f6f34d792764c4bc985dc0ae29\": container with ID starting with 690639b0b4db94b94d5d56158a9fcd34b4c7c8f6f34d792764c4bc985dc0ae29 not found: ID does not exist" Nov 04 04:01:44.420005 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.419867 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eed1e3d9-e514-4d8f-b1d3-851453900699" path="/var/lib/kubelet/pods/eed1e3d9-e514-4d8f-b1d3-851453900699/volumes" Nov 04 04:01:44.420795 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.420770 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl"] Nov 04 04:01:44.440356 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.440311 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" podUID="28e01b06-4f41-4ec9-b351-6145f5d0bd2a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 04:01:44.461687 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.461617 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" podUID="ba949fa3-73d8-4ae0-b586-b001b8988986" containerName="main" containerID="cri-o://258fc19481ee4d7f679307d12ecfb252055323661032b8ecaacf365987b9c478" gracePeriod=45 Nov 04 04:01:44.486465 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.485117 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" podUID="f939b53c-8938-4bb4-b334-7e329d1c366d" containerName="main" containerID="cri-o://11f4b7ea62ded68b45eaed7ec2a9b4c165492c148879a70cfc494362c7fb7095" gracePeriod=45 Nov 04 04:01:44.617565 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.613805 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" podUID="ba949fa3-73d8-4ae0-b586-b001b8988986" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": dial tcp 10.132.0.80:8000: connect: connection refused" Nov 04 04:01:44.620394 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.618368 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:44.622797 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.621625 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:44.683002 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.682359 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:01:44.803009 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.796536 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:44.809583 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.804410 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:44.809583 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.806162 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:01:44.823490 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.823461 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" Nov 04 04:01:44.871336 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.871304 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9ae98de-5794-48e0-ae47-d93b38c29ab0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ae98de-5794-48e0-ae47-d93b38c29ab0-openshift-service-ca\") pod \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\" (UID: \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\") " Nov 04 04:01:44.871563 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.871545 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9ae98de-5794-48e0-ae47-d93b38c29ab0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9ae98de-5794-48e0-ae47-d93b38c29ab0-maas-api-tls\") pod \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\" (UID: \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\") " Nov 04 04:01:44.871690 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.871678 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9ae98de-5794-48e0-ae47-d93b38c29ab0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9ae98de-5794-48e0-ae47-d93b38c29ab0-metrics-certs\") pod \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\" (UID: \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\") " Nov 04 04:01:44.871813 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.871801 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9ae98de-5794-48e0-ae47-d93b38c29ab0-kube-api-access-jhdcw\" (UniqueName: \"kubernetes.io/projected/b9ae98de-5794-48e0-ae47-d93b38c29ab0-kube-api-access-jhdcw\") pod \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\" (UID: \"b9ae98de-5794-48e0-ae47-d93b38c29ab0\") " Nov 04 04:01:44.873130 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.873107 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ae98de-5794-48e0-ae47-d93b38c29ab0-openshift-service-ca" pod "b9ae98de-5794-48e0-ae47-d93b38c29ab0" (UID: "b9ae98de-5794-48e0-ae47-d93b38c29ab0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:01:44.886160 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.886036 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9ae98de-5794-48e0-ae47-d93b38c29ab0-maas-api-tls" pod "b9ae98de-5794-48e0-ae47-d93b38c29ab0" (UID: "b9ae98de-5794-48e0-ae47-d93b38c29ab0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:01:44.886601 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.886553 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9ae98de-5794-48e0-ae47-d93b38c29ab0-metrics-certs" pod "b9ae98de-5794-48e0-ae47-d93b38c29ab0" (UID: "b9ae98de-5794-48e0-ae47-d93b38c29ab0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:01:44.891519 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.888646 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ae98de-5794-48e0-ae47-d93b38c29ab0-kube-api-access-jhdcw" pod "b9ae98de-5794-48e0-ae47-d93b38c29ab0" (UID: "b9ae98de-5794-48e0-ae47-d93b38c29ab0"). InnerVolumeSpecName "kube-api-access-jhdcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:01:44.973967 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.973431 4966 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9ae98de-5794-48e0-ae47-d93b38c29ab0-metrics-certs\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:44.973967 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.973467 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhdcw\" (UniqueName: \"kubernetes.io/projected/b9ae98de-5794-48e0-ae47-d93b38c29ab0-kube-api-access-jhdcw\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:44.973967 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.973484 4966 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ae98de-5794-48e0-ae47-d93b38c29ab0-openshift-service-ca\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:44.973967 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.973497 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9ae98de-5794-48e0-ae47-d93b38c29ab0-maas-api-tls\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:44.987629 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:44.987595 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:45.003063 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.001396 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:45.003063 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.001780 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:45.003421 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.003363 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:01:45.061383 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.061354 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 04:01:45.175868 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.175836 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f939b53c-8938-4bb4-b334-7e329d1c366d-kube-api-access-m4bxh\" (UniqueName: \"kubernetes.io/projected/f939b53c-8938-4bb4-b334-7e329d1c366d-kube-api-access-m4bxh\") pod \"f939b53c-8938-4bb4-b334-7e329d1c366d\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " Nov 04 04:01:45.176105 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.175914 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba949fa3-73d8-4ae0-b586-b001b8988986-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba949fa3-73d8-4ae0-b586-b001b8988986-tls-certs\") pod \"ba949fa3-73d8-4ae0-b586-b001b8988986\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " Nov 04 04:01:45.176105 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.175973 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-model-cache\") pod \"ba949fa3-73d8-4ae0-b586-b001b8988986\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " Nov 04 04:01:45.176105 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.176011 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-model-cache\") pod \"f939b53c-8938-4bb4-b334-7e329d1c366d\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " Nov 04 04:01:45.176105 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.176040 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-dshm\") pod \"f939b53c-8938-4bb4-b334-7e329d1c366d\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " Nov 04 04:01:45.176332 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.176188 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f939b53c-8938-4bb4-b334-7e329d1c366d-tls-certs\" (UniqueName: \"kubernetes.io/secret/f939b53c-8938-4bb4-b334-7e329d1c366d-tls-certs\") pod \"f939b53c-8938-4bb4-b334-7e329d1c366d\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " Nov 04 04:01:45.176332 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.176223 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba949fa3-73d8-4ae0-b586-b001b8988986-kube-api-access-szbcp\" (UniqueName: \"kubernetes.io/projected/ba949fa3-73d8-4ae0-b586-b001b8988986-kube-api-access-szbcp\") pod \"ba949fa3-73d8-4ae0-b586-b001b8988986\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " Nov 04 04:01:45.176332 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.176255 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-dshm\") pod \"ba949fa3-73d8-4ae0-b586-b001b8988986\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " Nov 04 04:01:45.176332 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.176291 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-model-cache" pod "f939b53c-8938-4bb4-b334-7e329d1c366d" (UID: "f939b53c-8938-4bb4-b334-7e329d1c366d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:45.176332 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.176317 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-home\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-home\") pod \"ba949fa3-73d8-4ae0-b586-b001b8988986\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " Nov 04 04:01:45.176580 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.176295 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-model-cache" pod "ba949fa3-73d8-4ae0-b586-b001b8988986" (UID: "ba949fa3-73d8-4ae0-b586-b001b8988986"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:45.176580 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.176377 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-tmp-dir\") pod \"f939b53c-8938-4bb4-b334-7e329d1c366d\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " Nov 04 04:01:45.176580 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.176428 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-tmp-dir\") pod \"ba949fa3-73d8-4ae0-b586-b001b8988986\" (UID: \"ba949fa3-73d8-4ae0-b586-b001b8988986\") " Nov 04 04:01:45.176580 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.176493 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-home\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-home\") pod \"f939b53c-8938-4bb4-b334-7e329d1c366d\" (UID: \"f939b53c-8938-4bb4-b334-7e329d1c366d\") " Nov 04 04:01:45.176782 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.176593 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-tmp-dir" pod "f939b53c-8938-4bb4-b334-7e329d1c366d" (UID: "f939b53c-8938-4bb4-b334-7e329d1c366d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:45.177651 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.177041 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-model-cache\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:45.177651 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.177067 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-model-cache\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:45.177651 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.177066 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-home" pod "ba949fa3-73d8-4ae0-b586-b001b8988986" (UID: "ba949fa3-73d8-4ae0-b586-b001b8988986"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:45.177651 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.177082 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-tmp-dir\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:45.177651 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.177521 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-tmp-dir" pod "ba949fa3-73d8-4ae0-b586-b001b8988986" (UID: "ba949fa3-73d8-4ae0-b586-b001b8988986"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:45.178026 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.177720 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-home" pod "f939b53c-8938-4bb4-b334-7e329d1c366d" (UID: "f939b53c-8938-4bb4-b334-7e329d1c366d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:45.179273 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.179226 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f939b53c-8938-4bb4-b334-7e329d1c366d-tls-certs" pod "f939b53c-8938-4bb4-b334-7e329d1c366d" (UID: "f939b53c-8938-4bb4-b334-7e329d1c366d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:01:45.179273 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.179235 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba949fa3-73d8-4ae0-b586-b001b8988986-kube-api-access-szbcp" pod "ba949fa3-73d8-4ae0-b586-b001b8988986" (UID: "ba949fa3-73d8-4ae0-b586-b001b8988986"). InnerVolumeSpecName "kube-api-access-szbcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:01:45.179273 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.179255 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-dshm" pod "ba949fa3-73d8-4ae0-b586-b001b8988986" (UID: "ba949fa3-73d8-4ae0-b586-b001b8988986"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:45.179711 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.179682 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-dshm" pod "f939b53c-8938-4bb4-b334-7e329d1c366d" (UID: "f939b53c-8938-4bb4-b334-7e329d1c366d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:45.180418 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.180391 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f939b53c-8938-4bb4-b334-7e329d1c366d-kube-api-access-m4bxh" pod "f939b53c-8938-4bb4-b334-7e329d1c366d" (UID: "f939b53c-8938-4bb4-b334-7e329d1c366d"). InnerVolumeSpecName "kube-api-access-m4bxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:01:45.180533 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.180452 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba949fa3-73d8-4ae0-b586-b001b8988986-tls-certs" pod "ba949fa3-73d8-4ae0-b586-b001b8988986" (UID: "ba949fa3-73d8-4ae0-b586-b001b8988986"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:01:45.278790 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.277858 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f939b53c-8938-4bb4-b334-7e329d1c366d-tls-certs\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:45.278790 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.277894 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szbcp\" (UniqueName: \"kubernetes.io/projected/ba949fa3-73d8-4ae0-b586-b001b8988986-kube-api-access-szbcp\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:45.278790 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.277910 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-dshm\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:45.278790 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.277955 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-home\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:45.278790 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.277970 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba949fa3-73d8-4ae0-b586-b001b8988986-tmp-dir\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:45.278790 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.277984 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-home\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:45.278790 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.277997 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4bxh\" (UniqueName: \"kubernetes.io/projected/f939b53c-8938-4bb4-b334-7e329d1c366d-kube-api-access-m4bxh\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:45.278790 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.278011 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba949fa3-73d8-4ae0-b586-b001b8988986-tls-certs\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:45.278790 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.278053 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f939b53c-8938-4bb4-b334-7e329d1c366d-dshm\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:45.278790 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.278331 4966 generic.go:356] "Generic (PLEG): container finished" podID="ba949fa3-73d8-4ae0-b586-b001b8988986" containerID="258fc19481ee4d7f679307d12ecfb252055323661032b8ecaacf365987b9c478" exitCode=0 Nov 04 04:01:45.278790 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.278386 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" Nov 04 04:01:45.278790 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.278410 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" event={"ID":"ba949fa3-73d8-4ae0-b586-b001b8988986","Type":"ContainerDied","Data":"258fc19481ee4d7f679307d12ecfb252055323661032b8ecaacf365987b9c478"} Nov 04 04:01:45.278790 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.278457 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7" event={"ID":"ba949fa3-73d8-4ae0-b586-b001b8988986","Type":"ContainerDied","Data":"9642bf1a29d64756b0d5ca15cbb656176ca0f2b60d146106a9dc4d07919a8960"} Nov 04 04:01:45.278790 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.278484 4966 scope.go:122] "RemoveContainer" containerID="258fc19481ee4d7f679307d12ecfb252055323661032b8ecaacf365987b9c478" Nov 04 04:01:45.283117 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.283095 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" Nov 04 04:01:45.287232 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.287171 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" event={"ID":"f939b53c-8938-4bb4-b334-7e329d1c366d","Type":"ContainerDied","Data":"11f4b7ea62ded68b45eaed7ec2a9b4c165492c148879a70cfc494362c7fb7095"} Nov 04 04:01:45.287232 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.287209 4966 generic.go:356] "Generic (PLEG): container finished" podID="f939b53c-8938-4bb4-b334-7e329d1c366d" containerID="11f4b7ea62ded68b45eaed7ec2a9b4c165492c148879a70cfc494362c7fb7095" exitCode=0 Nov 04 04:01:45.287412 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.287305 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" event={"ID":"f939b53c-8938-4bb4-b334-7e329d1c366d","Type":"ContainerDied","Data":"272d475a7cd384b96eb29e69ce5d92983ca782b797fdd3602130443ac070ef99"} Nov 04 04:01:45.295510 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.295448 4966 generic.go:356] "Generic (PLEG): container finished" podID="b9ae98de-5794-48e0-ae47-d93b38c29ab0" containerID="c097693692277ef812df5dfec02dd807044547f22510556c39546bc51d62fe4d" exitCode=0 Nov 04 04:01:45.296230 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.296214 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" Nov 04 04:01:45.309076 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.307427 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" event={"ID":"b9ae98de-5794-48e0-ae47-d93b38c29ab0","Type":"ContainerDied","Data":"c097693692277ef812df5dfec02dd807044547f22510556c39546bc51d62fe4d"} Nov 04 04:01:45.309076 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.307502 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf" event={"ID":"b9ae98de-5794-48e0-ae47-d93b38c29ab0","Type":"ContainerDied","Data":"53bebdf68303a5107e5f55fd143b0ed86a7c9908d872f1ac5fcd1eedc8dd3a34"} Nov 04 04:01:45.309076 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.308968 4966 scope.go:122] "RemoveContainer" containerID="258fc19481ee4d7f679307d12ecfb252055323661032b8ecaacf365987b9c478" Nov 04 04:01:45.309836 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:01:45.309806 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"258fc19481ee4d7f679307d12ecfb252055323661032b8ecaacf365987b9c478\": container with ID starting with 258fc19481ee4d7f679307d12ecfb252055323661032b8ecaacf365987b9c478 not found: ID does not exist" containerID="258fc19481ee4d7f679307d12ecfb252055323661032b8ecaacf365987b9c478" Nov 04 04:01:45.309961 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.309846 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258fc19481ee4d7f679307d12ecfb252055323661032b8ecaacf365987b9c478"} err="failed to get container status \"258fc19481ee4d7f679307d12ecfb252055323661032b8ecaacf365987b9c478\": rpc error: code = NotFound desc = could not find container \"258fc19481ee4d7f679307d12ecfb252055323661032b8ecaacf365987b9c478\": container with ID starting with 258fc19481ee4d7f679307d12ecfb252055323661032b8ecaacf365987b9c478 not found: ID does not exist" Nov 04 04:01:45.309961 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.309868 4966 scope.go:122] "RemoveContainer" containerID="11f4b7ea62ded68b45eaed7ec2a9b4c165492c148879a70cfc494362c7fb7095" Nov 04 04:01:45.335792 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.335736 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7"] Nov 04 04:01:45.340391 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.340372 4966 scope.go:122] "RemoveContainer" containerID="11f4b7ea62ded68b45eaed7ec2a9b4c165492c148879a70cfc494362c7fb7095" Nov 04 04:01:45.341009 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:01:45.340983 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f4b7ea62ded68b45eaed7ec2a9b4c165492c148879a70cfc494362c7fb7095\": container with ID starting with 11f4b7ea62ded68b45eaed7ec2a9b4c165492c148879a70cfc494362c7fb7095 not found: ID does not exist" containerID="11f4b7ea62ded68b45eaed7ec2a9b4c165492c148879a70cfc494362c7fb7095" Nov 04 04:01:45.341172 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.341018 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f4b7ea62ded68b45eaed7ec2a9b4c165492c148879a70cfc494362c7fb7095"} err="failed to get container status \"11f4b7ea62ded68b45eaed7ec2a9b4c165492c148879a70cfc494362c7fb7095\": rpc error: code = NotFound desc = could not find container \"11f4b7ea62ded68b45eaed7ec2a9b4c165492c148879a70cfc494362c7fb7095\": container with ID starting with 11f4b7ea62ded68b45eaed7ec2a9b4c165492c148879a70cfc494362c7fb7095 not found: ID does not exist" Nov 04 04:01:45.341172 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.341042 4966 scope.go:122] "RemoveContainer" containerID="c097693692277ef812df5dfec02dd807044547f22510556c39546bc51d62fe4d" Nov 04 04:01:45.341693 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.341668 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-64c20d/distinct-2-w4-64c20d-kserve-75dcfd896f-dnzd7"] Nov 04 04:01:45.369641 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.367093 4966 scope.go:122] "RemoveContainer" containerID="c097693692277ef812df5dfec02dd807044547f22510556c39546bc51d62fe4d" Nov 04 04:01:45.369641 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:01:45.369266 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c097693692277ef812df5dfec02dd807044547f22510556c39546bc51d62fe4d\": container with ID starting with c097693692277ef812df5dfec02dd807044547f22510556c39546bc51d62fe4d not found: ID does not exist" containerID="c097693692277ef812df5dfec02dd807044547f22510556c39546bc51d62fe4d" Nov 04 04:01:45.369641 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.369303 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c097693692277ef812df5dfec02dd807044547f22510556c39546bc51d62fe4d"} err="failed to get container status \"c097693692277ef812df5dfec02dd807044547f22510556c39546bc51d62fe4d\": rpc error: code = NotFound desc = could not find container \"c097693692277ef812df5dfec02dd807044547f22510556c39546bc51d62fe4d\": container with ID starting with c097693692277ef812df5dfec02dd807044547f22510556c39546bc51d62fe4d not found: ID does not exist" Nov 04 04:01:45.371020 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.370945 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh"] Nov 04 04:01:45.375856 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.375823 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh"] Nov 04 04:01:45.383862 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.383840 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf"] Nov 04 04:01:45.394222 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.394195 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-56a44f-68649fc44b-4bcwf"] Nov 04 04:01:45.939400 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:45.939285 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-64c20d/premium-w4-64c20d-kserve-9598d8bf5-cw5rh" podUID="f939b53c-8938-4bb4-b334-7e329d1c366d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 04:01:46.341338 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:46.341306 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7f1c31b-8f96-4c56-ab05-8e22896d6533" path="/var/lib/kubelet/pods/a7f1c31b-8f96-4c56-ab05-8e22896d6533/volumes" Nov 04 04:01:46.342192 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:46.341980 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9ae98de-5794-48e0-ae47-d93b38c29ab0" path="/var/lib/kubelet/pods/b9ae98de-5794-48e0-ae47-d93b38c29ab0/volumes" Nov 04 04:01:46.342492 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:46.342470 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba949fa3-73d8-4ae0-b586-b001b8988986" path="/var/lib/kubelet/pods/ba949fa3-73d8-4ae0-b586-b001b8988986/volumes" Nov 04 04:01:46.343261 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:46.343240 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f939b53c-8938-4bb4-b334-7e329d1c366d" path="/var/lib/kubelet/pods/f939b53c-8938-4bb4-b334-7e329d1c366d/volumes" Nov 04 04:01:50.490610 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:50.490577 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6"] Nov 04 04:01:50.504357 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:50.504325 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st"] Nov 04 04:01:51.544946 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:51.529335 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" podUID="76911349-4357-4a41-9218-a6077aed29bc" containerName="main" containerID="cri-o://41f19ff74a4c8cc49968fa9cb6ac1473c42345e7b091695e16670d85bd629daa" gracePeriod=45 Nov 04 04:01:51.986706 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:51.986666 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24"] Nov 04 04:01:51.987799 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:51.987757 4966 cpu_manager.go:414] "containerMap: removing container" podUID="eed1e3d9-e514-4d8f-b1d3-851453900699" containerName="main" Nov 04 04:01:51.987969 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:51.987954 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eed1e3d9-e514-4d8f-b1d3-851453900699" containerName="main" Nov 04 04:01:51.989415 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:51.988368 4966 cpu_manager.go:414] "containerMap: removing container" podUID="ba949fa3-73d8-4ae0-b586-b001b8988986" containerName="main" Nov 04 04:01:51.989415 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:51.988386 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba949fa3-73d8-4ae0-b586-b001b8988986" containerName="main" Nov 04 04:01:51.989415 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:51.988407 4966 cpu_manager.go:414] "containerMap: removing container" podUID="b9ae98de-5794-48e0-ae47-d93b38c29ab0" containerName="maas-api" Nov 04 04:01:51.989415 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:51.988414 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9ae98de-5794-48e0-ae47-d93b38c29ab0" containerName="maas-api" Nov 04 04:01:51.989415 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:51.988439 4966 cpu_manager.go:414] "containerMap: removing container" podUID="a7f1c31b-8f96-4c56-ab05-8e22896d6533" containerName="istio-proxy" Nov 04 04:01:51.989415 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:51.988446 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f1c31b-8f96-4c56-ab05-8e22896d6533" containerName="istio-proxy" Nov 04 04:01:51.989415 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:51.988482 4966 cpu_manager.go:414] "containerMap: removing container" podUID="f939b53c-8938-4bb4-b334-7e329d1c366d" containerName="main" Nov 04 04:01:51.989415 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:51.988488 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f939b53c-8938-4bb4-b334-7e329d1c366d" containerName="main" Nov 04 04:01:51.989415 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:51.988679 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="eed1e3d9-e514-4d8f-b1d3-851453900699" containerName="main" Nov 04 04:01:51.989415 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:51.988696 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba949fa3-73d8-4ae0-b586-b001b8988986" containerName="main" Nov 04 04:01:51.989415 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:51.988707 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9ae98de-5794-48e0-ae47-d93b38c29ab0" containerName="maas-api" Nov 04 04:01:51.989415 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:51.988717 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f1c31b-8f96-4c56-ab05-8e22896d6533" containerName="istio-proxy" Nov 04 04:01:51.989415 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:51.988743 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f939b53c-8938-4bb4-b334-7e329d1c366d" containerName="main" Nov 04 04:01:52.047389 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.047354 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24"] Nov 04 04:01:52.048527 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.048503 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.082088 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.071157 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 04 04:01:52.082088 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.081854 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 04 04:01:52.082088 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.081902 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 04 04:01:52.082088 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.082008 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qvwgz\"" Nov 04 04:01:52.235780 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.235222 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9747b332-91da-4e7c-ae71-700ac67567f9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.235780 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.235334 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9747b332-91da-4e7c-ae71-700ac67567f9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.235780 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.235374 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9747b332-91da-4e7c-ae71-700ac67567f9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.235780 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.235448 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9747b332-91da-4e7c-ae71-700ac67567f9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.235780 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.235472 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz8bj\" (UniqueName: \"kubernetes.io/projected/9747b332-91da-4e7c-ae71-700ac67567f9-kube-api-access-gz8bj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.235780 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.235497 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9747b332-91da-4e7c-ae71-700ac67567f9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.235780 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.235525 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9747b332-91da-4e7c-ae71-700ac67567f9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.236806 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.236742 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:01:52.385826 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.350647 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9747b332-91da-4e7c-ae71-700ac67567f9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.385826 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.350779 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9747b332-91da-4e7c-ae71-700ac67567f9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.385826 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.350816 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9747b332-91da-4e7c-ae71-700ac67567f9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.385826 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.350901 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9747b332-91da-4e7c-ae71-700ac67567f9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.385826 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.350941 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz8bj\" (UniqueName: \"kubernetes.io/projected/9747b332-91da-4e7c-ae71-700ac67567f9-kube-api-access-gz8bj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.385826 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.350970 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9747b332-91da-4e7c-ae71-700ac67567f9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.385826 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.351001 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9747b332-91da-4e7c-ae71-700ac67567f9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.405321 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.398551 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9747b332-91da-4e7c-ae71-700ac67567f9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.526764 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.456013 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-home\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-home\") pod \"76911349-4357-4a41-9218-a6077aed29bc\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " Nov 04 04:01:52.526764 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.456134 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76911349-4357-4a41-9218-a6077aed29bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/76911349-4357-4a41-9218-a6077aed29bc-tls-certs\") pod \"76911349-4357-4a41-9218-a6077aed29bc\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " Nov 04 04:01:52.526764 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.456167 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-tmp-dir\") pod \"76911349-4357-4a41-9218-a6077aed29bc\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " Nov 04 04:01:52.526764 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.456304 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76911349-4357-4a41-9218-a6077aed29bc-kube-api-access-6hkrx\" (UniqueName: \"kubernetes.io/projected/76911349-4357-4a41-9218-a6077aed29bc-kube-api-access-6hkrx\") pod \"76911349-4357-4a41-9218-a6077aed29bc\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " Nov 04 04:01:52.526764 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.456331 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-dshm\") pod \"76911349-4357-4a41-9218-a6077aed29bc\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " Nov 04 04:01:52.526764 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.456356 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-model-cache\") pod \"76911349-4357-4a41-9218-a6077aed29bc\" (UID: \"76911349-4357-4a41-9218-a6077aed29bc\") " Nov 04 04:01:52.526764 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.462226 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-model-cache" pod "76911349-4357-4a41-9218-a6077aed29bc" (UID: "76911349-4357-4a41-9218-a6077aed29bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:52.526764 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.468009 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-tmp-dir" pod "76911349-4357-4a41-9218-a6077aed29bc" (UID: "76911349-4357-4a41-9218-a6077aed29bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:52.526764 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.503080 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-home" pod "76911349-4357-4a41-9218-a6077aed29bc" (UID: "76911349-4357-4a41-9218-a6077aed29bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:52.568082 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.558890 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-model-cache\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:52.568082 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.558939 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-home\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:52.568082 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.558959 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-tmp-dir\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:52.573724 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.573645 4966 generic.go:356] "Generic (PLEG): container finished" podID="76911349-4357-4a41-9218-a6077aed29bc" containerID="41f19ff74a4c8cc49968fa9cb6ac1473c42345e7b091695e16670d85bd629daa" exitCode=0 Nov 04 04:01:52.577127 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.576456 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" event={"ID":"76911349-4357-4a41-9218-a6077aed29bc","Type":"ContainerDied","Data":"41f19ff74a4c8cc49968fa9cb6ac1473c42345e7b091695e16670d85bd629daa"} Nov 04 04:01:52.577127 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.576536 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" event={"ID":"76911349-4357-4a41-9218-a6077aed29bc","Type":"ContainerDied","Data":"ea02093753988cd232bce5b4623b9bef6112a8c3ed0ee2acd22723ae9e2ac1d8"} Nov 04 04:01:52.577127 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.576566 4966 scope.go:122] "RemoveContainer" containerID="41f19ff74a4c8cc49968fa9cb6ac1473c42345e7b091695e16670d85bd629daa" Nov 04 04:01:52.577127 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.576735 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8" Nov 04 04:01:52.643962 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.636965 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz8bj\" (UniqueName: \"kubernetes.io/projected/9747b332-91da-4e7c-ae71-700ac67567f9-kube-api-access-gz8bj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.671771 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.671398 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9747b332-91da-4e7c-ae71-700ac67567f9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.682227 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.682150 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9747b332-91da-4e7c-ae71-700ac67567f9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.698961 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.693434 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9747b332-91da-4e7c-ae71-700ac67567f9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.698961 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.694365 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9747b332-91da-4e7c-ae71-700ac67567f9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.767055 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.766996 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-dshm" pod "76911349-4357-4a41-9218-a6077aed29bc" (UID: "76911349-4357-4a41-9218-a6077aed29bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:01:52.767757 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.767714 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76911349-4357-4a41-9218-a6077aed29bc-tls-certs" pod "76911349-4357-4a41-9218-a6077aed29bc" (UID: "76911349-4357-4a41-9218-a6077aed29bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:01:52.768097 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.768056 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76911349-4357-4a41-9218-a6077aed29bc-kube-api-access-6hkrx" pod "76911349-4357-4a41-9218-a6077aed29bc" (UID: "76911349-4357-4a41-9218-a6077aed29bc"). InnerVolumeSpecName "kube-api-access-6hkrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:01:52.841565 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.823739 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hkrx\" (UniqueName: \"kubernetes.io/projected/76911349-4357-4a41-9218-a6077aed29bc-kube-api-access-6hkrx\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:52.841565 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.823780 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76911349-4357-4a41-9218-a6077aed29bc-dshm\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:52.841565 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.823797 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76911349-4357-4a41-9218-a6077aed29bc-tls-certs\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:52.841565 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.833017 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9747b332-91da-4e7c-ae71-700ac67567f9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ck24\" (UID: \"9747b332-91da-4e7c-ae71-700ac67567f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:52.938697 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.933795 4966 scope.go:122] "RemoveContainer" containerID="41f19ff74a4c8cc49968fa9cb6ac1473c42345e7b091695e16670d85bd629daa" Nov 04 04:01:52.938799 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:01:52.938780 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f19ff74a4c8cc49968fa9cb6ac1473c42345e7b091695e16670d85bd629daa\": container with ID starting with 41f19ff74a4c8cc49968fa9cb6ac1473c42345e7b091695e16670d85bd629daa not found: ID does not exist" containerID="41f19ff74a4c8cc49968fa9cb6ac1473c42345e7b091695e16670d85bd629daa" Nov 04 04:01:52.938862 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.938812 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f19ff74a4c8cc49968fa9cb6ac1473c42345e7b091695e16670d85bd629daa"} err="failed to get container status \"41f19ff74a4c8cc49968fa9cb6ac1473c42345e7b091695e16670d85bd629daa\": rpc error: code = NotFound desc = could not find container \"41f19ff74a4c8cc49968fa9cb6ac1473c42345e7b091695e16670d85bd629daa\": container with ID starting with 41f19ff74a4c8cc49968fa9cb6ac1473c42345e7b091695e16670d85bd629daa not found: ID does not exist" Nov 04 04:01:52.995944 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:52.981362 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8"] Nov 04 04:01:53.020112 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:53.011557 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4afe0e/distinct-w0-4afe0e-kserve-5f94d449bd-blnp8"] Nov 04 04:01:53.100002 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:53.098127 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:53.496349 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:53.496257 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24"] Nov 04 04:01:53.585606 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:53.585564 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" event={"ID":"9747b332-91da-4e7c-ae71-700ac67567f9","Type":"ContainerStarted","Data":"442287c0b5563f31bd27878782a3f333564c8ff9df5ee9a50b98505a512b0e7b"} Nov 04 04:01:54.342219 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:54.342188 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76911349-4357-4a41-9218-a6077aed29bc" path="/var/lib/kubelet/pods/76911349-4357-4a41-9218-a6077aed29bc/volumes" Nov 04 04:01:54.387218 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:54.387177 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" podUID="28e01b06-4f41-4ec9-b351-6145f5d0bd2a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 04:01:55.491901 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:55.491854 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6" podUID="6626886d-db18-4cd8-ae04-23e2b5841223" containerName="payload-processing" containerID="cri-o://5229455eca23d4b2d5b8da2f671625a6ac1064d75932fdbd28f7f26ccf15e285" gracePeriod=25 Nov 04 04:01:55.505400 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:55.505335 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" podUID="c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1" containerName="payload-pre-processing" containerID="cri-o://211e4acc080cca8f065a7480f31f1cacbd7d72d478286d335b9588a69e33c5ba" gracePeriod=25 Nov 04 04:01:55.800011 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:55.799979 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6" Nov 04 04:01:55.871533 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:55.871500 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6626886d-db18-4cd8-ae04-23e2b5841223-kube-api-access-h56g9\" (UniqueName: \"kubernetes.io/projected/6626886d-db18-4cd8-ae04-23e2b5841223-kube-api-access-h56g9\") pod \"6626886d-db18-4cd8-ae04-23e2b5841223\" (UID: \"6626886d-db18-4cd8-ae04-23e2b5841223\") " Nov 04 04:01:55.871700 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:55.871625 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6626886d-db18-4cd8-ae04-23e2b5841223-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6626886d-db18-4cd8-ae04-23e2b5841223-plugins-config-volume\") pod \"6626886d-db18-4cd8-ae04-23e2b5841223\" (UID: \"6626886d-db18-4cd8-ae04-23e2b5841223\") " Nov 04 04:01:55.872583 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:55.872532 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6626886d-db18-4cd8-ae04-23e2b5841223-plugins-config-volume" pod "6626886d-db18-4cd8-ae04-23e2b5841223" (UID: "6626886d-db18-4cd8-ae04-23e2b5841223"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:01:55.874310 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:55.874286 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6626886d-db18-4cd8-ae04-23e2b5841223-kube-api-access-h56g9" pod "6626886d-db18-4cd8-ae04-23e2b5841223" (UID: "6626886d-db18-4cd8-ae04-23e2b5841223"). InnerVolumeSpecName "kube-api-access-h56g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:01:55.973691 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:55.973647 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h56g9\" (UniqueName: \"kubernetes.io/projected/6626886d-db18-4cd8-ae04-23e2b5841223-kube-api-access-h56g9\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:55.973691 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:55.973687 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6626886d-db18-4cd8-ae04-23e2b5841223-plugins-config-volume\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:56.613412 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:56.613322 4966 generic.go:356] "Generic (PLEG): container finished" podID="6626886d-db18-4cd8-ae04-23e2b5841223" containerID="5229455eca23d4b2d5b8da2f671625a6ac1064d75932fdbd28f7f26ccf15e285" exitCode=0 Nov 04 04:01:56.613412 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:56.613376 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6" Nov 04 04:01:56.613984 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:56.613404 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6" event={"ID":"6626886d-db18-4cd8-ae04-23e2b5841223","Type":"ContainerDied","Data":"5229455eca23d4b2d5b8da2f671625a6ac1064d75932fdbd28f7f26ccf15e285"} Nov 04 04:01:56.613984 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:56.613479 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6" event={"ID":"6626886d-db18-4cd8-ae04-23e2b5841223","Type":"ContainerDied","Data":"0396385859e38a83f25989136938461400fb785d88c3abfa7b940b129b0629a9"} Nov 04 04:01:56.613984 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:56.613511 4966 scope.go:122] "RemoveContainer" containerID="5229455eca23d4b2d5b8da2f671625a6ac1064d75932fdbd28f7f26ccf15e285" Nov 04 04:01:56.625301 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:56.625277 4966 scope.go:122] "RemoveContainer" containerID="5229455eca23d4b2d5b8da2f671625a6ac1064d75932fdbd28f7f26ccf15e285" Nov 04 04:01:56.625599 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:01:56.625572 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5229455eca23d4b2d5b8da2f671625a6ac1064d75932fdbd28f7f26ccf15e285\": container with ID starting with 5229455eca23d4b2d5b8da2f671625a6ac1064d75932fdbd28f7f26ccf15e285 not found: ID does not exist" containerID="5229455eca23d4b2d5b8da2f671625a6ac1064d75932fdbd28f7f26ccf15e285" Nov 04 04:01:56.625714 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:56.625610 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5229455eca23d4b2d5b8da2f671625a6ac1064d75932fdbd28f7f26ccf15e285"} err="failed to get container status \"5229455eca23d4b2d5b8da2f671625a6ac1064d75932fdbd28f7f26ccf15e285\": rpc error: code = NotFound desc = could not find container \"5229455eca23d4b2d5b8da2f671625a6ac1064d75932fdbd28f7f26ccf15e285\": container with ID starting with 5229455eca23d4b2d5b8da2f671625a6ac1064d75932fdbd28f7f26ccf15e285 not found: ID does not exist" Nov 04 04:01:56.634035 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:56.634008 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6"] Nov 04 04:01:56.637822 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:56.637801 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6"] Nov 04 04:01:57.152365 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.152327 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:01:57.152482 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.152407 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 04 04:01:57.267968 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.267942 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" Nov 04 04:01:57.289890 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.289599 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e01b06-4f41-4ec9-b351-6145f5d0bd2a-kube-api-access-svn79\" (UniqueName: \"kubernetes.io/projected/28e01b06-4f41-4ec9-b351-6145f5d0bd2a-kube-api-access-svn79\") pod \"28e01b06-4f41-4ec9-b351-6145f5d0bd2a\" (UID: \"28e01b06-4f41-4ec9-b351-6145f5d0bd2a\") " Nov 04 04:01:57.289890 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.289667 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28e01b06-4f41-4ec9-b351-6145f5d0bd2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e01b06-4f41-4ec9-b351-6145f5d0bd2a-plugins-config-volume\") pod \"28e01b06-4f41-4ec9-b351-6145f5d0bd2a\" (UID: \"28e01b06-4f41-4ec9-b351-6145f5d0bd2a\") " Nov 04 04:01:57.290535 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.290511 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e01b06-4f41-4ec9-b351-6145f5d0bd2a-plugins-config-volume" pod "28e01b06-4f41-4ec9-b351-6145f5d0bd2a" (UID: "28e01b06-4f41-4ec9-b351-6145f5d0bd2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:01:57.292637 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.292609 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e01b06-4f41-4ec9-b351-6145f5d0bd2a-kube-api-access-svn79" pod "28e01b06-4f41-4ec9-b351-6145f5d0bd2a" (UID: "28e01b06-4f41-4ec9-b351-6145f5d0bd2a"). InnerVolumeSpecName "kube-api-access-svn79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:01:57.391200 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.391170 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svn79\" (UniqueName: \"kubernetes.io/projected/28e01b06-4f41-4ec9-b351-6145f5d0bd2a-kube-api-access-svn79\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:57.391200 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.391196 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e01b06-4f41-4ec9-b351-6145f5d0bd2a-plugins-config-volume\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:01:57.623442 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.623399 4966 generic.go:356] "Generic (PLEG): container finished" podID="28e01b06-4f41-4ec9-b351-6145f5d0bd2a" containerID="641b44eef565e6d3d1eccc4b26631e2eef89384deddacce49e0b2c14471ab5ac" exitCode=0 Nov 04 04:01:57.623864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.623504 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" event={"ID":"28e01b06-4f41-4ec9-b351-6145f5d0bd2a","Type":"ContainerDied","Data":"641b44eef565e6d3d1eccc4b26631e2eef89384deddacce49e0b2c14471ab5ac"} Nov 04 04:01:57.623864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.623559 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" event={"ID":"28e01b06-4f41-4ec9-b351-6145f5d0bd2a","Type":"ContainerDied","Data":"7254bd2a02bee8fb4c6e3360cae55e8dbbffd2116845bd33b8119dad733309fb"} Nov 04 04:01:57.623864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.623590 4966 scope.go:122] "RemoveContainer" containerID="641b44eef565e6d3d1eccc4b26631e2eef89384deddacce49e0b2c14471ab5ac" Nov 04 04:01:57.623864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.623697 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp" Nov 04 04:01:57.632699 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.632668 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" event={"ID":"9747b332-91da-4e7c-ae71-700ac67567f9","Type":"ContainerStarted","Data":"001f01cd6019939de9699b9b209657a4189cfd8e8ef6e10cb67fd8ab7c7a485b"} Nov 04 04:01:57.632886 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.632869 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:01:57.639813 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.639797 4966 scope.go:122] "RemoveContainer" containerID="641b44eef565e6d3d1eccc4b26631e2eef89384deddacce49e0b2c14471ab5ac" Nov 04 04:01:57.640139 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:01:57.640117 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"641b44eef565e6d3d1eccc4b26631e2eef89384deddacce49e0b2c14471ab5ac\": container with ID starting with 641b44eef565e6d3d1eccc4b26631e2eef89384deddacce49e0b2c14471ab5ac not found: ID does not exist" containerID="641b44eef565e6d3d1eccc4b26631e2eef89384deddacce49e0b2c14471ab5ac" Nov 04 04:01:57.640210 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.640146 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641b44eef565e6d3d1eccc4b26631e2eef89384deddacce49e0b2c14471ab5ac"} err="failed to get container status \"641b44eef565e6d3d1eccc4b26631e2eef89384deddacce49e0b2c14471ab5ac\": rpc error: code = NotFound desc = could not find container \"641b44eef565e6d3d1eccc4b26631e2eef89384deddacce49e0b2c14471ab5ac\": container with ID starting with 641b44eef565e6d3d1eccc4b26631e2eef89384deddacce49e0b2c14471ab5ac not found: ID does not exist" Nov 04 04:01:57.661246 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.658958 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" podStartSLOduration=3.023628398 podStartE2EDuration="6.658942293s" totalImagesPullingTime="3.635313895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:01:57.655628322 +0000 UTC m=+761.456574164" watchObservedRunningTime="2026-11-04 04:01:57.658942293 +0000 UTC m=+761.459888098" Nov 04 04:01:57.678673 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.678637 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp"] Nov 04 04:01:57.681174 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:57.681150 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp"] Nov 04 04:01:58.340937 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:58.340844 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e01b06-4f41-4ec9-b351-6145f5d0bd2a" path="/var/lib/kubelet/pods/28e01b06-4f41-4ec9-b351-6145f5d0bd2a/volumes" Nov 04 04:01:58.341485 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:58.341467 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6626886d-db18-4cd8-ae04-23e2b5841223" path="/var/lib/kubelet/pods/6626886d-db18-4cd8-ae04-23e2b5841223/volumes" Nov 04 04:01:58.644221 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:01:58.644141 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24" Nov 04 04:02:02.456354 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:02.456319 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp"] Nov 04 04:02:02.456835 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:02.456553 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" podUID="ff721376-eba2-4a06-b1b9-48e5875d9c05" containerName="maas-api" containerID="cri-o://6f35ee0162dd28be19156dc2285c3ed675ee82b4ff785245527a2c7c384665a7" gracePeriod=30 Nov 04 04:02:02.950616 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:02.950588 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" Nov 04 04:02:03.053273 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.053183 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff721376-eba2-4a06-b1b9-48e5875d9c05-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff721376-eba2-4a06-b1b9-48e5875d9c05-metrics-certs\") pod \"ff721376-eba2-4a06-b1b9-48e5875d9c05\" (UID: \"ff721376-eba2-4a06-b1b9-48e5875d9c05\") " Nov 04 04:02:03.053425 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.053277 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff721376-eba2-4a06-b1b9-48e5875d9c05-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff721376-eba2-4a06-b1b9-48e5875d9c05-openshift-service-ca\") pod \"ff721376-eba2-4a06-b1b9-48e5875d9c05\" (UID: \"ff721376-eba2-4a06-b1b9-48e5875d9c05\") " Nov 04 04:02:03.053425 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.053317 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff721376-eba2-4a06-b1b9-48e5875d9c05-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff721376-eba2-4a06-b1b9-48e5875d9c05-maas-api-tls\") pod \"ff721376-eba2-4a06-b1b9-48e5875d9c05\" (UID: \"ff721376-eba2-4a06-b1b9-48e5875d9c05\") " Nov 04 04:02:03.053425 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.053370 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff721376-eba2-4a06-b1b9-48e5875d9c05-kube-api-access-7sg5d\" (UniqueName: \"kubernetes.io/projected/ff721376-eba2-4a06-b1b9-48e5875d9c05-kube-api-access-7sg5d\") pod \"ff721376-eba2-4a06-b1b9-48e5875d9c05\" (UID: \"ff721376-eba2-4a06-b1b9-48e5875d9c05\") " Nov 04 04:02:03.053666 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.053636 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff721376-eba2-4a06-b1b9-48e5875d9c05-openshift-service-ca" pod "ff721376-eba2-4a06-b1b9-48e5875d9c05" (UID: "ff721376-eba2-4a06-b1b9-48e5875d9c05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:02:03.053795 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.053778 4966 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff721376-eba2-4a06-b1b9-48e5875d9c05-openshift-service-ca\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:03.055651 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.055614 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff721376-eba2-4a06-b1b9-48e5875d9c05-maas-api-tls" pod "ff721376-eba2-4a06-b1b9-48e5875d9c05" (UID: "ff721376-eba2-4a06-b1b9-48e5875d9c05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:02:03.055743 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.055685 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff721376-eba2-4a06-b1b9-48e5875d9c05-metrics-certs" pod "ff721376-eba2-4a06-b1b9-48e5875d9c05" (UID: "ff721376-eba2-4a06-b1b9-48e5875d9c05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:02:03.055743 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.055732 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff721376-eba2-4a06-b1b9-48e5875d9c05-kube-api-access-7sg5d" pod "ff721376-eba2-4a06-b1b9-48e5875d9c05" (UID: "ff721376-eba2-4a06-b1b9-48e5875d9c05"). InnerVolumeSpecName "kube-api-access-7sg5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:02:03.155434 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.155396 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff721376-eba2-4a06-b1b9-48e5875d9c05-maas-api-tls\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:03.155434 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.155429 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sg5d\" (UniqueName: \"kubernetes.io/projected/ff721376-eba2-4a06-b1b9-48e5875d9c05-kube-api-access-7sg5d\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:03.155434 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.155440 4966 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff721376-eba2-4a06-b1b9-48e5875d9c05-metrics-certs\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:03.367221 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.367179 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" podUID="c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 04:02:03.736796 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.736721 4966 generic.go:356] "Generic (PLEG): container finished" podID="ff721376-eba2-4a06-b1b9-48e5875d9c05" containerID="6f35ee0162dd28be19156dc2285c3ed675ee82b4ff785245527a2c7c384665a7" exitCode=0 Nov 04 04:02:03.736796 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.736780 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" Nov 04 04:02:03.737308 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.736801 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" event={"ID":"ff721376-eba2-4a06-b1b9-48e5875d9c05","Type":"ContainerDied","Data":"6f35ee0162dd28be19156dc2285c3ed675ee82b4ff785245527a2c7c384665a7"} Nov 04 04:02:03.737308 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.736858 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp" event={"ID":"ff721376-eba2-4a06-b1b9-48e5875d9c05","Type":"ContainerDied","Data":"87f8c2a203c511f857bc2ad59e7ae1d663ce96d6f16b9cb3b3e650ede6ecf549"} Nov 04 04:02:03.737308 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.736890 4966 scope.go:122] "RemoveContainer" containerID="6f35ee0162dd28be19156dc2285c3ed675ee82b4ff785245527a2c7c384665a7" Nov 04 04:02:03.748614 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.748588 4966 scope.go:122] "RemoveContainer" containerID="6f35ee0162dd28be19156dc2285c3ed675ee82b4ff785245527a2c7c384665a7" Nov 04 04:02:03.748887 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:02:03.748856 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f35ee0162dd28be19156dc2285c3ed675ee82b4ff785245527a2c7c384665a7\": container with ID starting with 6f35ee0162dd28be19156dc2285c3ed675ee82b4ff785245527a2c7c384665a7 not found: ID does not exist" containerID="6f35ee0162dd28be19156dc2285c3ed675ee82b4ff785245527a2c7c384665a7" Nov 04 04:02:03.748981 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.748895 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f35ee0162dd28be19156dc2285c3ed675ee82b4ff785245527a2c7c384665a7"} err="failed to get container status \"6f35ee0162dd28be19156dc2285c3ed675ee82b4ff785245527a2c7c384665a7\": rpc error: code = NotFound desc = could not find container \"6f35ee0162dd28be19156dc2285c3ed675ee82b4ff785245527a2c7c384665a7\": container with ID starting with 6f35ee0162dd28be19156dc2285c3ed675ee82b4ff785245527a2c7c384665a7 not found: ID does not exist" Nov 04 04:02:03.763895 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.763870 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp"] Nov 04 04:02:03.766458 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:03.766435 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4afe0e-576b8784c4-mvhbp"] Nov 04 04:02:04.340662 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:04.340628 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff721376-eba2-4a06-b1b9-48e5875d9c05" path="/var/lib/kubelet/pods/ff721376-eba2-4a06-b1b9-48e5875d9c05/volumes" Nov 04 04:02:05.206212 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:05.206176 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8"] Nov 04 04:02:05.206703 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:05.206419 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" podUID="8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" containerName="istio-proxy" containerID="cri-o://2a2401eb60d2d683a092245c56a74c2a7240c450835813bfe3eccf29ebd2848e" gracePeriod=30 Nov 04 04:02:05.214553 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:05.214521 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn"] Nov 04 04:02:05.214812 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:05.214757 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" podUID="2060984f-5a85-4790-b19c-3e6fff9016ae" containerName="istio-proxy" containerID="cri-o://8cbf6ce60026f824f759368dba5101dc9d8a3391668343b29bc4b96600760d17" gracePeriod=30 Nov 04 04:02:05.676602 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:02:05.676566 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 04:02:07.569903 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:07.569859 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv"] Nov 04 04:02:07.570476 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:07.570184 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" podUID="388ae503-75e8-4507-b6c2-9e4229c67128" containerName="istio-proxy" containerID="cri-o://629ecbdb3ffcb8bbbbc4b67f44531a847a4c37dd509310da4d1f2b752a547841" gracePeriod=30 Nov 04 04:02:07.586346 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:07.586300 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf"] Nov 04 04:02:07.586565 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:07.586543 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" podUID="8ed1dacd-75d6-4e61-94bb-d7fcced94f85" containerName="istio-proxy" containerID="cri-o://274b102a5dfea0df566582f92ebdc426fa29428b46ded6491344cd3251e0e5ce" gracePeriod=30 Nov 04 04:02:10.487105 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.487080 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:02:10.533205 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.533180 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:02:10.549148 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.549113 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-workload-certs\") pod \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " Nov 04 04:02:10.549303 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.549184 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istiod-ca-cert\") pod \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " Nov 04 04:02:10.549303 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.549267 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-token\" (UniqueName: \"kubernetes.io/projected/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-token\") pod \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " Nov 04 04:02:10.549416 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.549315 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-credential-socket\") pod \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " Nov 04 04:02:10.549416 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.549354 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-kube-api-access-nr8j8\" (UniqueName: \"kubernetes.io/projected/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-kube-api-access-nr8j8\") pod \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " Nov 04 04:02:10.549525 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.549428 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-data\") pod \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " Nov 04 04:02:10.549525 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.549430 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-workload-certs" pod "8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" (UID: "8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:10.549624 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.549608 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-workload-socket\") pod \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " Nov 04 04:02:10.549696 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.549671 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-credential-socket" pod "8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" (UID: "8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:10.549840 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.549812 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-data" pod "8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" (UID: "8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:10.550064 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.549887 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-podinfo\") pod \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " Nov 04 04:02:10.550064 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.549914 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istiod-ca-cert" pod "8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" (UID: "8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:02:10.550064 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.550032 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-workload-socket" pod "8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" (UID: "8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:10.550382 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.550296 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-envoy\") pod \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\" (UID: \"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4\") " Nov 04 04:02:10.551178 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.551155 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-workload-certs\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:10.551386 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.551365 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istiod-ca-cert\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:10.551499 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.551485 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-credential-socket\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:10.551585 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.551574 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-data\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:10.551679 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.551667 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-workload-socket\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:10.555094 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.555069 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-podinfo" pod "8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" (UID: "8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 04:02:10.555233 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.555109 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-kube-api-access-nr8j8" pod "8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" (UID: "8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4"). InnerVolumeSpecName "kube-api-access-nr8j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:02:10.555233 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.555173 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-token" pod "8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" (UID: "8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:02:10.556099 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.556078 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-envoy" pod "8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" (UID: "8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:10.653136 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.653105 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-envoy\") pod \"2060984f-5a85-4790-b19c-3e6fff9016ae\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " Nov 04 04:02:10.653328 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.653147 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-credential-socket\") pod \"2060984f-5a85-4790-b19c-3e6fff9016ae\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " Nov 04 04:02:10.653328 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.653172 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-token\" (UniqueName: \"kubernetes.io/projected/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-token\") pod \"2060984f-5a85-4790-b19c-3e6fff9016ae\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " Nov 04 04:02:10.653328 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.653212 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-podinfo\") pod \"2060984f-5a85-4790-b19c-3e6fff9016ae\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " Nov 04 04:02:10.653328 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.653248 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2060984f-5a85-4790-b19c-3e6fff9016ae-kube-api-access-gl27c\" (UniqueName: \"kubernetes.io/projected/2060984f-5a85-4790-b19c-3e6fff9016ae-kube-api-access-gl27c\") pod \"2060984f-5a85-4790-b19c-3e6fff9016ae\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " Nov 04 04:02:10.653328 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.653267 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-workload-certs\") pod \"2060984f-5a85-4790-b19c-3e6fff9016ae\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " Nov 04 04:02:10.653328 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.653323 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-data\") pod \"2060984f-5a85-4790-b19c-3e6fff9016ae\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " Nov 04 04:02:10.653651 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.653385 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-workload-socket\") pod \"2060984f-5a85-4790-b19c-3e6fff9016ae\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " Nov 04 04:02:10.653651 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.653417 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2060984f-5a85-4790-b19c-3e6fff9016ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2060984f-5a85-4790-b19c-3e6fff9016ae-istiod-ca-cert\") pod \"2060984f-5a85-4790-b19c-3e6fff9016ae\" (UID: \"2060984f-5a85-4790-b19c-3e6fff9016ae\") " Nov 04 04:02:10.653651 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.653486 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-credential-socket" pod "2060984f-5a85-4790-b19c-3e6fff9016ae" (UID: "2060984f-5a85-4790-b19c-3e6fff9016ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:10.653651 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.653597 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-workload-certs" pod "2060984f-5a85-4790-b19c-3e6fff9016ae" (UID: "2060984f-5a85-4790-b19c-3e6fff9016ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:10.653651 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.653626 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-data" pod "2060984f-5a85-4790-b19c-3e6fff9016ae" (UID: "2060984f-5a85-4790-b19c-3e6fff9016ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:10.653894 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.653712 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-workload-socket" pod "2060984f-5a85-4790-b19c-3e6fff9016ae" (UID: "2060984f-5a85-4790-b19c-3e6fff9016ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:10.653956 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.653888 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2060984f-5a85-4790-b19c-3e6fff9016ae-istiod-ca-cert" pod "2060984f-5a85-4790-b19c-3e6fff9016ae" (UID: "2060984f-5a85-4790-b19c-3e6fff9016ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:02:10.653991 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.653955 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-podinfo\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:10.653991 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.653975 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-envoy\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:10.654068 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.653992 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-data\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:10.654068 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.654006 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-workload-socket\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:10.654068 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.654020 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-credential-socket\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:10.654068 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.654034 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-istio-token\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:10.654068 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.654049 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr8j8\" (UniqueName: \"kubernetes.io/projected/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4-kube-api-access-nr8j8\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:10.654068 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.654063 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-workload-certs\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:10.655463 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.655443 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-token" pod "2060984f-5a85-4790-b19c-3e6fff9016ae" (UID: "2060984f-5a85-4790-b19c-3e6fff9016ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:02:10.655693 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.655677 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-envoy" pod "2060984f-5a85-4790-b19c-3e6fff9016ae" (UID: "2060984f-5a85-4790-b19c-3e6fff9016ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:10.655761 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.655696 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2060984f-5a85-4790-b19c-3e6fff9016ae-kube-api-access-gl27c" pod "2060984f-5a85-4790-b19c-3e6fff9016ae" (UID: "2060984f-5a85-4790-b19c-3e6fff9016ae"). InnerVolumeSpecName "kube-api-access-gl27c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:02:10.655879 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.655864 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-podinfo" pod "2060984f-5a85-4790-b19c-3e6fff9016ae" (UID: "2060984f-5a85-4790-b19c-3e6fff9016ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 04:02:10.755318 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.755233 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-envoy\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:10.755318 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.755262 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-token\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:10.755318 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.755271 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2060984f-5a85-4790-b19c-3e6fff9016ae-istio-podinfo\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:10.755318 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.755280 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl27c\" (UniqueName: \"kubernetes.io/projected/2060984f-5a85-4790-b19c-3e6fff9016ae-kube-api-access-gl27c\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:10.755318 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.755290 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2060984f-5a85-4790-b19c-3e6fff9016ae-istiod-ca-cert\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:10.789334 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.789300 4966 generic.go:356] "Generic (PLEG): container finished" podID="8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" containerID="2a2401eb60d2d683a092245c56a74c2a7240c450835813bfe3eccf29ebd2848e" exitCode=0 Nov 04 04:02:10.789517 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.789349 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" Nov 04 04:02:10.789517 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.789379 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" event={"ID":"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4","Type":"ContainerDied","Data":"2a2401eb60d2d683a092245c56a74c2a7240c450835813bfe3eccf29ebd2848e"} Nov 04 04:02:10.789517 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.789453 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8" event={"ID":"8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4","Type":"ContainerDied","Data":"fb6d9a9f3030b1b87cc3afe4208a921e456ce31fafadc35ef4d497d19cc0e928"} Nov 04 04:02:10.789517 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.789493 4966 scope.go:122] "RemoveContainer" containerID="2a2401eb60d2d683a092245c56a74c2a7240c450835813bfe3eccf29ebd2848e" Nov 04 04:02:10.791036 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.791010 4966 generic.go:356] "Generic (PLEG): container finished" podID="2060984f-5a85-4790-b19c-3e6fff9016ae" containerID="8cbf6ce60026f824f759368dba5101dc9d8a3391668343b29bc4b96600760d17" exitCode=0 Nov 04 04:02:10.791180 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.791062 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" Nov 04 04:02:10.791180 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.791094 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" event={"ID":"2060984f-5a85-4790-b19c-3e6fff9016ae","Type":"ContainerDied","Data":"8cbf6ce60026f824f759368dba5101dc9d8a3391668343b29bc4b96600760d17"} Nov 04 04:02:10.791180 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.791140 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn" event={"ID":"2060984f-5a85-4790-b19c-3e6fff9016ae","Type":"ContainerDied","Data":"749904aafe752d4f7be5af6c402098a99a9a20e45ab67f7c0b06cbb597bf6ca5"} Nov 04 04:02:10.807198 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.807176 4966 scope.go:122] "RemoveContainer" containerID="2a2401eb60d2d683a092245c56a74c2a7240c450835813bfe3eccf29ebd2848e" Nov 04 04:02:10.807492 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:02:10.807469 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a2401eb60d2d683a092245c56a74c2a7240c450835813bfe3eccf29ebd2848e\": container with ID starting with 2a2401eb60d2d683a092245c56a74c2a7240c450835813bfe3eccf29ebd2848e not found: ID does not exist" containerID="2a2401eb60d2d683a092245c56a74c2a7240c450835813bfe3eccf29ebd2848e" Nov 04 04:02:10.807586 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.807502 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a2401eb60d2d683a092245c56a74c2a7240c450835813bfe3eccf29ebd2848e"} err="failed to get container status \"2a2401eb60d2d683a092245c56a74c2a7240c450835813bfe3eccf29ebd2848e\": rpc error: code = NotFound desc = could not find container \"2a2401eb60d2d683a092245c56a74c2a7240c450835813bfe3eccf29ebd2848e\": container with ID starting with 2a2401eb60d2d683a092245c56a74c2a7240c450835813bfe3eccf29ebd2848e not found: ID does not exist" Nov 04 04:02:10.807586 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.807526 4966 scope.go:122] "RemoveContainer" containerID="8cbf6ce60026f824f759368dba5101dc9d8a3391668343b29bc4b96600760d17" Nov 04 04:02:10.816713 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.816690 4966 scope.go:122] "RemoveContainer" containerID="8cbf6ce60026f824f759368dba5101dc9d8a3391668343b29bc4b96600760d17" Nov 04 04:02:10.817039 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:02:10.817009 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cbf6ce60026f824f759368dba5101dc9d8a3391668343b29bc4b96600760d17\": container with ID starting with 8cbf6ce60026f824f759368dba5101dc9d8a3391668343b29bc4b96600760d17 not found: ID does not exist" containerID="8cbf6ce60026f824f759368dba5101dc9d8a3391668343b29bc4b96600760d17" Nov 04 04:02:10.817126 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.817049 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cbf6ce60026f824f759368dba5101dc9d8a3391668343b29bc4b96600760d17"} err="failed to get container status \"8cbf6ce60026f824f759368dba5101dc9d8a3391668343b29bc4b96600760d17\": rpc error: code = NotFound desc = could not find container \"8cbf6ce60026f824f759368dba5101dc9d8a3391668343b29bc4b96600760d17\": container with ID starting with 8cbf6ce60026f824f759368dba5101dc9d8a3391668343b29bc4b96600760d17 not found: ID does not exist" Nov 04 04:02:10.871937 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.871891 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8"] Nov 04 04:02:10.875151 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.875124 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8"] Nov 04 04:02:10.912197 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.912164 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn"] Nov 04 04:02:10.916296 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:10.916259 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn"] Nov 04 04:02:12.340947 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.340878 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2060984f-5a85-4790-b19c-3e6fff9016ae" path="/var/lib/kubelet/pods/2060984f-5a85-4790-b19c-3e6fff9016ae/volumes" Nov 04 04:02:12.341578 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.341555 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" path="/var/lib/kubelet/pods/8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4/volumes" Nov 04 04:02:12.811449 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.811415 4966 generic.go:356] "Generic (PLEG): container finished" podID="8ed1dacd-75d6-4e61-94bb-d7fcced94f85" containerID="274b102a5dfea0df566582f92ebdc426fa29428b46ded6491344cd3251e0e5ce" exitCode=0 Nov 04 04:02:12.811648 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.811493 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" event={"ID":"8ed1dacd-75d6-4e61-94bb-d7fcced94f85","Type":"ContainerDied","Data":"274b102a5dfea0df566582f92ebdc426fa29428b46ded6491344cd3251e0e5ce"} Nov 04 04:02:12.813429 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.813404 4966 generic.go:356] "Generic (PLEG): container finished" podID="388ae503-75e8-4507-b6c2-9e4229c67128" containerID="629ecbdb3ffcb8bbbbc4b67f44531a847a4c37dd509310da4d1f2b752a547841" exitCode=0 Nov 04 04:02:12.813590 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.813570 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" event={"ID":"388ae503-75e8-4507-b6c2-9e4229c67128","Type":"ContainerDied","Data":"629ecbdb3ffcb8bbbbc4b67f44531a847a4c37dd509310da4d1f2b752a547841"} Nov 04 04:02:12.880330 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.880301 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:02:12.911555 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.911527 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:02:12.978685 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.978633 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-istio-data\") pod \"388ae503-75e8-4507-b6c2-9e4229c67128\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " Nov 04 04:02:12.978888 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.978739 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/388ae503-75e8-4507-b6c2-9e4229c67128-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388ae503-75e8-4507-b6c2-9e4229c67128-istiod-ca-cert\") pod \"388ae503-75e8-4507-b6c2-9e4229c67128\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " Nov 04 04:02:12.978888 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.978795 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-workload-socket\") pod \"388ae503-75e8-4507-b6c2-9e4229c67128\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " Nov 04 04:02:12.978888 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.978835 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-credential-socket\") pod \"388ae503-75e8-4507-b6c2-9e4229c67128\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " Nov 04 04:02:12.978888 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.978874 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388ae503-75e8-4507-b6c2-9e4229c67128-kube-api-access-44l7q\" (UniqueName: \"kubernetes.io/projected/388ae503-75e8-4507-b6c2-9e4229c67128-kube-api-access-44l7q\") pod \"388ae503-75e8-4507-b6c2-9e4229c67128\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " Nov 04 04:02:12.979121 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.978990 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-istio-envoy\") pod \"388ae503-75e8-4507-b6c2-9e4229c67128\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " Nov 04 04:02:12.979121 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.979003 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-istio-data" pod "388ae503-75e8-4507-b6c2-9e4229c67128" (UID: "388ae503-75e8-4507-b6c2-9e4229c67128"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:12.979121 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.979056 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388ae503-75e8-4507-b6c2-9e4229c67128-istio-token\" (UniqueName: \"kubernetes.io/projected/388ae503-75e8-4507-b6c2-9e4229c67128-istio-token\") pod \"388ae503-75e8-4507-b6c2-9e4229c67128\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " Nov 04 04:02:12.979276 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.979125 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-workload-certs\") pod \"388ae503-75e8-4507-b6c2-9e4229c67128\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " Nov 04 04:02:12.979276 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.979207 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/388ae503-75e8-4507-b6c2-9e4229c67128-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388ae503-75e8-4507-b6c2-9e4229c67128-istio-podinfo\") pod \"388ae503-75e8-4507-b6c2-9e4229c67128\" (UID: \"388ae503-75e8-4507-b6c2-9e4229c67128\") " Nov 04 04:02:12.979276 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.979213 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-workload-socket" pod "388ae503-75e8-4507-b6c2-9e4229c67128" (UID: "388ae503-75e8-4507-b6c2-9e4229c67128"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:12.979568 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.979527 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388ae503-75e8-4507-b6c2-9e4229c67128-istiod-ca-cert" pod "388ae503-75e8-4507-b6c2-9e4229c67128" (UID: "388ae503-75e8-4507-b6c2-9e4229c67128"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:02:12.979732 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.979682 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-istio-data\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:12.979732 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.979702 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388ae503-75e8-4507-b6c2-9e4229c67128-istiod-ca-cert\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:12.979732 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.979717 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-workload-socket\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:12.980516 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.980184 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-workload-certs" pod "388ae503-75e8-4507-b6c2-9e4229c67128" (UID: "388ae503-75e8-4507-b6c2-9e4229c67128"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:12.980516 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.980244 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-credential-socket" pod "388ae503-75e8-4507-b6c2-9e4229c67128" (UID: "388ae503-75e8-4507-b6c2-9e4229c67128"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:12.983991 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.983950 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/388ae503-75e8-4507-b6c2-9e4229c67128-istio-podinfo" pod "388ae503-75e8-4507-b6c2-9e4229c67128" (UID: "388ae503-75e8-4507-b6c2-9e4229c67128"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 04:02:12.984353 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.984305 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388ae503-75e8-4507-b6c2-9e4229c67128-kube-api-access-44l7q" pod "388ae503-75e8-4507-b6c2-9e4229c67128" (UID: "388ae503-75e8-4507-b6c2-9e4229c67128"). InnerVolumeSpecName "kube-api-access-44l7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:02:12.988717 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.988688 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-istio-envoy" pod "388ae503-75e8-4507-b6c2-9e4229c67128" (UID: "388ae503-75e8-4507-b6c2-9e4229c67128"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:12.989194 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:12.989154 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388ae503-75e8-4507-b6c2-9e4229c67128-istio-token" pod "388ae503-75e8-4507-b6c2-9e4229c67128" (UID: "388ae503-75e8-4507-b6c2-9e4229c67128"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:02:13.080244 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.080205 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-data\") pod \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " Nov 04 04:02:13.080428 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.080269 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-kube-api-access-shzjn\" (UniqueName: \"kubernetes.io/projected/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-kube-api-access-shzjn\") pod \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " Nov 04 04:02:13.080428 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.080327 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-workload-certs\") pod \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " Nov 04 04:02:13.080428 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.080344 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-credential-socket\") pod \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " Nov 04 04:02:13.080428 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.080375 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-token\" (UniqueName: \"kubernetes.io/projected/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-token\") pod \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " Nov 04 04:02:13.080633 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.080514 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istiod-ca-cert\") pod \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " Nov 04 04:02:13.080633 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.080602 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-workload-socket\") pod \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " Nov 04 04:02:13.080633 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.080603 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-workload-certs" pod "8ed1dacd-75d6-4e61-94bb-d7fcced94f85" (UID: "8ed1dacd-75d6-4e61-94bb-d7fcced94f85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:13.080781 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.080645 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-podinfo\") pod \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " Nov 04 04:02:13.080781 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.080698 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-credential-socket" pod "8ed1dacd-75d6-4e61-94bb-d7fcced94f85" (UID: "8ed1dacd-75d6-4e61-94bb-d7fcced94f85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:13.080781 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.080753 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-envoy\") pod \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\" (UID: \"8ed1dacd-75d6-4e61-94bb-d7fcced94f85\") " Nov 04 04:02:13.080936 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.080887 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-workload-socket" pod "8ed1dacd-75d6-4e61-94bb-d7fcced94f85" (UID: "8ed1dacd-75d6-4e61-94bb-d7fcced94f85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:13.080996 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.080979 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istiod-ca-cert" pod "8ed1dacd-75d6-4e61-94bb-d7fcced94f85" (UID: "8ed1dacd-75d6-4e61-94bb-d7fcced94f85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:02:13.081243 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.080625 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-data" pod "8ed1dacd-75d6-4e61-94bb-d7fcced94f85" (UID: "8ed1dacd-75d6-4e61-94bb-d7fcced94f85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:13.081345 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.081248 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-credential-socket\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:13.081345 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.081271 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44l7q\" (UniqueName: \"kubernetes.io/projected/388ae503-75e8-4507-b6c2-9e4229c67128-kube-api-access-44l7q\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:13.081345 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.081287 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-istio-envoy\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:13.081345 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.081302 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-workload-certs\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:13.081345 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.081317 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-credential-socket\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:13.081345 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.081332 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388ae503-75e8-4507-b6c2-9e4229c67128-istio-token\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:13.081651 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.081347 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istiod-ca-cert\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:13.081651 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.081365 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388ae503-75e8-4507-b6c2-9e4229c67128-workload-certs\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:13.081651 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.081378 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-workload-socket\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:13.081651 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.081391 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388ae503-75e8-4507-b6c2-9e4229c67128-istio-podinfo\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:13.082761 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.082733 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-kube-api-access-shzjn" pod "8ed1dacd-75d6-4e61-94bb-d7fcced94f85" (UID: "8ed1dacd-75d6-4e61-94bb-d7fcced94f85"). InnerVolumeSpecName "kube-api-access-shzjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:02:13.083113 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.083084 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-envoy" pod "8ed1dacd-75d6-4e61-94bb-d7fcced94f85" (UID: "8ed1dacd-75d6-4e61-94bb-d7fcced94f85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:13.083113 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.083099 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-podinfo" pod "8ed1dacd-75d6-4e61-94bb-d7fcced94f85" (UID: "8ed1dacd-75d6-4e61-94bb-d7fcced94f85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 04:02:13.083113 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.083095 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-token" pod "8ed1dacd-75d6-4e61-94bb-d7fcced94f85" (UID: "8ed1dacd-75d6-4e61-94bb-d7fcced94f85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:02:13.183069 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.182988 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-data\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:13.183069 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.183019 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shzjn\" (UniqueName: \"kubernetes.io/projected/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-kube-api-access-shzjn\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:13.183069 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.183029 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-token\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:13.183069 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.183038 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-podinfo\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:13.183069 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.183046 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed1dacd-75d6-4e61-94bb-d7fcced94f85-istio-envoy\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:13.366455 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.366419 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" podUID="c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 04:02:13.821515 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.821484 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" Nov 04 04:02:13.821688 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.821490 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf" event={"ID":"8ed1dacd-75d6-4e61-94bb-d7fcced94f85","Type":"ContainerDied","Data":"9ca910132520cbe85883377df3449b7affeee6132c2f1b59787baa87988f159b"} Nov 04 04:02:13.821688 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.821628 4966 scope.go:122] "RemoveContainer" containerID="274b102a5dfea0df566582f92ebdc426fa29428b46ded6491344cd3251e0e5ce" Nov 04 04:02:13.823408 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.823385 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" Nov 04 04:02:13.823522 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.823412 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv" event={"ID":"388ae503-75e8-4507-b6c2-9e4229c67128","Type":"ContainerDied","Data":"e9d6c8f9f27dbf389264bc096115adc61edda2c4217eaac89f109a8698cdbea6"} Nov 04 04:02:13.834905 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.834883 4966 scope.go:122] "RemoveContainer" containerID="629ecbdb3ffcb8bbbbc4b67f44531a847a4c37dd509310da4d1f2b752a547841" Nov 04 04:02:13.862792 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.862724 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf"] Nov 04 04:02:13.871178 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.871148 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf"] Nov 04 04:02:13.888825 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.888796 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv"] Nov 04 04:02:13.896382 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:13.896349 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv"] Nov 04 04:02:14.341456 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:14.341414 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="388ae503-75e8-4507-b6c2-9e4229c67128" path="/var/lib/kubelet/pods/388ae503-75e8-4507-b6c2-9e4229c67128/volumes" Nov 04 04:02:14.342078 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:14.342048 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ed1dacd-75d6-4e61-94bb-d7fcced94f85" path="/var/lib/kubelet/pods/8ed1dacd-75d6-4e61-94bb-d7fcced94f85/volumes" Nov 04 04:02:15.819974 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:15.817281 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd"] Nov 04 04:02:15.819974 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:15.817535 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" podUID="1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" containerName="istio-proxy" containerID="cri-o://1a84f817ae0b87d071668a645713431f560e0c76d3367f81bf18240a467e77a8" gracePeriod=30 Nov 04 04:02:15.850194 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:15.849161 4966 generic.go:356] "Generic (PLEG): container finished" podID="c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1" containerID="211e4acc080cca8f065a7480f31f1cacbd7d72d478286d335b9588a69e33c5ba" exitCode=0 Nov 04 04:02:15.850194 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:15.849242 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" event={"ID":"c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1","Type":"ContainerDied","Data":"211e4acc080cca8f065a7480f31f1cacbd7d72d478286d335b9588a69e33c5ba"} Nov 04 04:02:15.850194 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:15.849283 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" event={"ID":"c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1","Type":"ContainerDied","Data":"7b250f1effe41e03e2644980fe184778e43af5da537a8009cb424377f47b4ca1"} Nov 04 04:02:15.850194 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:15.849301 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b250f1effe41e03e2644980fe184778e43af5da537a8009cb424377f47b4ca1" Nov 04 04:02:15.850907 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:15.850883 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" Nov 04 04:02:16.020505 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:16.020463 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1-kube-api-access-kkk5p\" (UniqueName: \"kubernetes.io/projected/c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1-kube-api-access-kkk5p\") pod \"c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1\" (UID: \"c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1\") " Nov 04 04:02:16.020698 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:16.020532 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1-plugins-config-volume\") pod \"c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1\" (UID: \"c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1\") " Nov 04 04:02:16.021017 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:16.020983 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1-plugins-config-volume" pod "c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1" (UID: "c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:02:16.022755 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:16.022722 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1-kube-api-access-kkk5p" pod "c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1" (UID: "c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1"). InnerVolumeSpecName "kube-api-access-kkk5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:02:16.121759 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:16.121727 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkk5p\" (UniqueName: \"kubernetes.io/projected/c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1-kube-api-access-kkk5p\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:16.121759 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:16.121752 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1-plugins-config-volume\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:16.854073 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:16.854039 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st" Nov 04 04:02:16.887120 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:16.887085 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st"] Nov 04 04:02:16.895824 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:16.895795 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st"] Nov 04 04:02:17.641480 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:17.641448 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cbbff8c7f-crdpg_dec9abf1-8b69-46d3-b5da-cb0b552b8447/authorino/0.log" Nov 04 04:02:18.340725 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:18.340692 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1" path="/var/lib/kubelet/pods/c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1/volumes" Nov 04 04:02:21.092563 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.092537 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:02:21.273876 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.273788 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istiod-ca-cert\") pod \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " Nov 04 04:02:21.273876 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.273838 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-token\" (UniqueName: \"kubernetes.io/projected/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-token\") pod \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " Nov 04 04:02:21.274105 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.273885 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-data\") pod \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " Nov 04 04:02:21.274105 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.273948 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-credential-socket\") pod \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " Nov 04 04:02:21.274105 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.274015 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-kube-api-access-6h7rl\" (UniqueName: \"kubernetes.io/projected/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-kube-api-access-6h7rl\") pod \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " Nov 04 04:02:21.274105 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.274050 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-workload-certs\") pod \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " Nov 04 04:02:21.274298 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.274116 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-envoy\") pod \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " Nov 04 04:02:21.274298 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.274191 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-workload-socket\") pod \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " Nov 04 04:02:21.274298 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.274217 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istiod-ca-cert" pod "1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" (UID: "1e0e1ba5-30ca-49f3-be4b-2e5a474d036a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:02:21.274298 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.274239 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-data" pod "1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" (UID: "1e0e1ba5-30ca-49f3-be4b-2e5a474d036a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:21.274505 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.274435 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-podinfo\") pod \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\" (UID: \"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a\") " Nov 04 04:02:21.274505 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.274456 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-credential-socket" pod "1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" (UID: "1e0e1ba5-30ca-49f3-be4b-2e5a474d036a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:21.274505 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.274472 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-workload-certs" pod "1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" (UID: "1e0e1ba5-30ca-49f3-be4b-2e5a474d036a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:21.274656 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.274635 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-workload-socket" pod "1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" (UID: "1e0e1ba5-30ca-49f3-be4b-2e5a474d036a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:21.274899 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.274873 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-workload-socket\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:21.274899 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.274900 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istiod-ca-cert\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:21.275127 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.274948 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-data\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:21.275127 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.274965 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-credential-socket\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:21.275127 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.274979 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-workload-certs\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:21.276472 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.276446 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-envoy" pod "1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" (UID: "1e0e1ba5-30ca-49f3-be4b-2e5a474d036a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:02:21.276472 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.276451 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-token" pod "1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" (UID: "1e0e1ba5-30ca-49f3-be4b-2e5a474d036a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:02:21.276616 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.276500 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-podinfo" pod "1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" (UID: "1e0e1ba5-30ca-49f3-be4b-2e5a474d036a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 04:02:21.276616 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.276581 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-kube-api-access-6h7rl" pod "1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" (UID: "1e0e1ba5-30ca-49f3-be4b-2e5a474d036a"). InnerVolumeSpecName "kube-api-access-6h7rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:02:21.376479 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.376439 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-podinfo\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:21.376479 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.376472 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-token\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:21.376479 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.376482 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h7rl\" (UniqueName: \"kubernetes.io/projected/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-kube-api-access-6h7rl\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:21.376724 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.376492 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a-istio-envoy\") on node \"ip-10-0-138-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 04:02:21.884259 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.884224 4966 generic.go:356] "Generic (PLEG): container finished" podID="1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" containerID="1a84f817ae0b87d071668a645713431f560e0c76d3367f81bf18240a467e77a8" exitCode=0 Nov 04 04:02:21.884431 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.884267 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" Nov 04 04:02:21.884431 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.884309 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" event={"ID":"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a","Type":"ContainerDied","Data":"1a84f817ae0b87d071668a645713431f560e0c76d3367f81bf18240a467e77a8"} Nov 04 04:02:21.884431 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.884365 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd" event={"ID":"1e0e1ba5-30ca-49f3-be4b-2e5a474d036a","Type":"ContainerDied","Data":"dbe1b2e9823a1f7a82686dce8ba23c9926fc1c3e2c15dd87a9cd3f073cbed410"} Nov 04 04:02:21.884431 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.884389 4966 scope.go:122] "RemoveContainer" containerID="1a84f817ae0b87d071668a645713431f560e0c76d3367f81bf18240a467e77a8" Nov 04 04:02:21.894762 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.894735 4966 scope.go:122] "RemoveContainer" containerID="1a84f817ae0b87d071668a645713431f560e0c76d3367f81bf18240a467e77a8" Nov 04 04:02:21.895097 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:02:21.895074 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a84f817ae0b87d071668a645713431f560e0c76d3367f81bf18240a467e77a8\": container with ID starting with 1a84f817ae0b87d071668a645713431f560e0c76d3367f81bf18240a467e77a8 not found: ID does not exist" containerID="1a84f817ae0b87d071668a645713431f560e0c76d3367f81bf18240a467e77a8" Nov 04 04:02:21.895162 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.895112 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a84f817ae0b87d071668a645713431f560e0c76d3367f81bf18240a467e77a8"} err="failed to get container status \"1a84f817ae0b87d071668a645713431f560e0c76d3367f81bf18240a467e77a8\": rpc error: code = NotFound desc = could not find container \"1a84f817ae0b87d071668a645713431f560e0c76d3367f81bf18240a467e77a8\": container with ID starting with 1a84f817ae0b87d071668a645713431f560e0c76d3367f81bf18240a467e77a8 not found: ID does not exist" Nov 04 04:02:21.910596 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.910564 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd"] Nov 04 04:02:21.913228 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:21.913205 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd"] Nov 04 04:02:22.341059 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:22.341025 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" path="/var/lib/kubelet/pods/1e0e1ba5-30ca-49f3-be4b-2e5a474d036a/volumes" Nov 04 04:02:29.059527 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:29.059492 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-58vj8_545fb734-bf40-43f8-a19d-0fe82840ef71/manager/0.log" Nov 04 04:02:29.240809 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:29.240776 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sdgx5_f2f40e58-66ab-4754-bd77-ed8e3d20a818/manager/0.log" Nov 04 04:02:29.704945 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:29.704897 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55cc4d9668-f6wtl_c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59/manager/0.log" Nov 04 04:02:29.929600 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:29.929571 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-2m46k_bffcd9a5-5579-4223-ad60-f1c5a862645f/server/0.log" Nov 04 04:02:30.114506 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:30.114472 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-xdp9r_06eeac89-43b5-4e21-bd3c-e480330635e8/manager/0.log" Nov 04 04:02:31.214469 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:31.214433 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5df874889d-sr95h_9e6b9d46-036a-4177-8583-c06b1bbba58b/maas-api/0.log" Nov 04 04:02:35.566700 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:35.566665 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cbbff8c7f-crdpg_dec9abf1-8b69-46d3-b5da-cb0b552b8447/authorino/0.log" Nov 04 04:02:35.676910 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:02:35.676871 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 04:02:36.100296 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:36.100266 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vb95x_5cb703ee-a590-4816-819d-2fb98480affa/kuadrant-console-plugin/0.log" Nov 04 04:02:37.938126 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:37.938086 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5ck24_9747b332-91da-4e7c-ae71-700ac67567f9/discovery/0.log" Nov 04 04:02:38.494265 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:38.494235 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bd469f65f-x7t8v_f376642e-e719-4112-858d-5dc9575b1236/kube-auth-proxy/0.log" Nov 04 04:02:38.675046 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:38.675016 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-4tpt8_177f8aff-8b11-4891-9648-9c7aeb9305b7/istio-proxy/0.log" Nov 04 04:02:39.391176 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:39.391142 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-n5x92_aa513df7-058f-4dbc-8743-f0b1e3f5e088/istio-proxy/0.log" Nov 04 04:02:40.105862 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:40.105830 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf_12af8c07-513e-4b8c-8172-4aebb9731f66/istio-proxy/0.log" Nov 04 04:02:40.282326 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:40.282297 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-lqlbm_c5ea3aa4-a31f-430e-ab69-31d81f975218/payload-pre-processing/0.log" Nov 04 04:02:40.444263 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:40.444184 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-sv8xh_94562f01-24b5-4df4-ae94-967d5a116a9c/payload-processing/0.log" Nov 04 04:02:40.621746 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:40.621710 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78ccf8f879-s4sx4_1b81b9bc-4eb5-48a7-98e1-6c7a98117f57/router/0.log" Nov 04 04:02:41.750130 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:41.750096 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv_6f7b4b6c-19ba-445e-bd66-5292b0a28b0e/main/0.log" Nov 04 04:02:41.926933 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:41.926890 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-vjthh_ba477b57-6755-44f3-b42f-595bcf525d4a/main/0.log" Nov 04 04:02:42.273375 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:42.273350 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-6gs5b_3871362e-919f-4375-8030-cc85243036a4/main/0.log" Nov 04 04:02:42.795287 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:42.795260 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c_6353ea56-85d9-422d-90e1-580b08d86d5f/main/0.log" Nov 04 04:02:44.016213 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:44.016186 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cbbff8c7f-crdpg_dec9abf1-8b69-46d3-b5da-cb0b552b8447/authorino/0.log" Nov 04 04:02:53.139689 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:53.139656 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-58vj8_545fb734-bf40-43f8-a19d-0fe82840ef71/manager/0.log" Nov 04 04:02:53.304219 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:53.304185 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sdgx5_f2f40e58-66ab-4754-bd77-ed8e3d20a818/manager/0.log" Nov 04 04:02:53.724369 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:53.724320 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55cc4d9668-f6wtl_c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59/manager/0.log" Nov 04 04:02:53.913788 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:53.913756 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-2m46k_bffcd9a5-5579-4223-ad60-f1c5a862645f/server/0.log" Nov 04 04:02:54.088568 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:54.088538 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-xdp9r_06eeac89-43b5-4e21-bd3c-e480330635e8/manager/0.log" Nov 04 04:02:54.784414 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:54.784388 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5df874889d-sr95h_9e6b9d46-036a-4177-8583-c06b1bbba58b/maas-api/0.log" Nov 04 04:02:58.377407 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:58.377372 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cbbff8c7f-crdpg_dec9abf1-8b69-46d3-b5da-cb0b552b8447/authorino/0.log" Nov 04 04:02:58.915012 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:02:58.914984 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vb95x_5cb703ee-a590-4816-819d-2fb98480affa/kuadrant-console-plugin/0.log" Nov 04 04:03:00.222788 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:00.222752 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5ck24_9747b332-91da-4e7c-ae71-700ac67567f9/discovery/0.log" Nov 04 04:03:00.757174 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:00.757138 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bd469f65f-x7t8v_f376642e-e719-4112-858d-5dc9575b1236/kube-auth-proxy/0.log" Nov 04 04:03:00.933256 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:00.933222 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-4tpt8_177f8aff-8b11-4891-9648-9c7aeb9305b7/istio-proxy/0.log" Nov 04 04:03:01.678588 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:01.678533 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-n5x92_aa513df7-058f-4dbc-8743-f0b1e3f5e088/istio-proxy/0.log" Nov 04 04:03:02.420542 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:02.420504 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf_12af8c07-513e-4b8c-8172-4aebb9731f66/istio-proxy/0.log" Nov 04 04:03:02.583654 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:02.583627 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-lqlbm_c5ea3aa4-a31f-430e-ab69-31d81f975218/payload-pre-processing/0.log" Nov 04 04:03:02.747138 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:02.747057 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-sv8xh_94562f01-24b5-4df4-ae94-967d5a116a9c/payload-processing/0.log" Nov 04 04:03:02.910688 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:02.910662 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78ccf8f879-s4sx4_1b81b9bc-4eb5-48a7-98e1-6c7a98117f57/router/0.log" Nov 04 04:03:03.410151 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:03.410120 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-9wvwv_6f7b4b6c-19ba-445e-bd66-5292b0a28b0e/main/0.log" Nov 04 04:03:03.589983 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:03.589951 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-vjthh_ba477b57-6755-44f3-b42f-595bcf525d4a/main/0.log" Nov 04 04:03:03.924679 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:03.924644 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-6gs5b_3871362e-919f-4375-8030-cc85243036a4/main/0.log" Nov 04 04:03:04.457390 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:04.457359 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-kwt8c_6353ea56-85d9-422d-90e1-580b08d86d5f/main/0.log" Nov 04 04:03:05.677024 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:03:05.676953 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 04:03:26.745903 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:26.745869 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55cc4d9668-f6wtl_c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59/manager/0.log" Nov 04 04:03:26.970318 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:26.970279 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55cc4d9668-f6wtl_c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59/manager/0.log" Nov 04 04:03:27.677179 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:27.677146 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-sv8xh_94562f01-24b5-4df4-ae94-967d5a116a9c/payload-processing/0.log" Nov 04 04:03:27.866914 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:27.866887 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-sv8xh_94562f01-24b5-4df4-ae94-967d5a116a9c/payload-processing/0.log" Nov 04 04:03:35.676496 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:03:35.676443 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 04:03:58.716117 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:58.716083 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55cc4d9668-f6wtl_c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59/manager/0.log" Nov 04 04:03:58.966984 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:58.966880 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55cc4d9668-f6wtl_c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59/manager/0.log" Nov 04 04:03:59.672499 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:59.672467 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-sv8xh_94562f01-24b5-4df4-ae94-967d5a116a9c/payload-processing/0.log" Nov 04 04:03:59.840084 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:03:59.840043 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-sv8xh_94562f01-24b5-4df4-ae94-967d5a116a9c/payload-processing/0.log" Nov 04 04:04:03.517595 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:03.517564 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-59fsx_26bb99fd-d223-4642-b147-b22bebd00835/global-pull-secret-syncer/0.log" Nov 04 04:04:03.696025 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:03.695997 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-8v2sd_27d72728-5623-41e0-bee5-a09937f9f87a/connection-checker/0.log" Nov 04 04:04:03.725579 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:03.725544 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-qq74s_1f0a14d4-9771-4aa2-a6bd-87f5f73ee931/connection-checker/0.log" Nov 04 04:04:03.755135 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:03.755110 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-wc5zp_1c2ac1f5-6e08-48e3-ac83-a55f9181f4be/connection-checker/0.log" Nov 04 04:04:03.879720 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:03.879686 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jvqw2_93149be5-0d1f-4b08-b33e-d2e43bac726c/konnectivity-agent/0.log" Nov 04 04:04:03.961647 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:03.961620 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-9.us-east-2.compute.internal_37008f3824659b029a90a4ae33f9a950/haproxy/0.log" Nov 04 04:04:05.676939 ip-10-0-138-9 kubenswrapper[4966]: E1104 04:04:05.676892 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zfjhp" containerName="extractor" Nov 04 04:04:10.281557 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:10.281522 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cbbff8c7f-crdpg_dec9abf1-8b69-46d3-b5da-cb0b552b8447/authorino/0.log" Nov 04 04:04:10.421046 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:10.421017 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vb95x_5cb703ee-a590-4816-819d-2fb98480affa/kuadrant-console-plugin/0.log" Nov 04 04:04:12.561237 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:12.561201 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-qmh6r_0df87f5d-31f4-4571-82b6-fe75c434ff27/cluster-monitoring-operator/0.log" Nov 04 04:04:12.760911 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:12.760882 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mnzz5_a2706ae5-dd36-4a7c-ba89-cf0a497f92b0/node-exporter/0.log" Nov 04 04:04:12.799825 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:12.799796 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mnzz5_a2706ae5-dd36-4a7c-ba89-cf0a497f92b0/kube-rbac-proxy/0.log" Nov 04 04:04:12.828040 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:12.827961 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mnzz5_a2706ae5-dd36-4a7c-ba89-cf0a497f92b0/init-textfile/0.log" Nov 04 04:04:13.467274 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.467243 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-5nfh9_ae821c43-9b3a-408a-8090-82ff08946607/prometheus-operator-admission-webhook/0.log" Nov 04 04:04:13.737934 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.737833 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b"] Nov 04 04:04:13.738435 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738416 4966 cpu_manager.go:414] "containerMap: removing container" podUID="1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" containerName="istio-proxy" Nov 04 04:04:13.738481 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738438 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" containerName="istio-proxy" Nov 04 04:04:13.738481 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738459 4966 cpu_manager.go:414] "containerMap: removing container" podUID="ff721376-eba2-4a06-b1b9-48e5875d9c05" containerName="maas-api" Nov 04 04:04:13.738481 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738466 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff721376-eba2-4a06-b1b9-48e5875d9c05" containerName="maas-api" Nov 04 04:04:13.738574 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738484 4966 cpu_manager.go:414] "containerMap: removing container" podUID="8ed1dacd-75d6-4e61-94bb-d7fcced94f85" containerName="istio-proxy" Nov 04 04:04:13.738574 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738491 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed1dacd-75d6-4e61-94bb-d7fcced94f85" containerName="istio-proxy" Nov 04 04:04:13.738574 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738501 4966 cpu_manager.go:414] "containerMap: removing container" podUID="388ae503-75e8-4507-b6c2-9e4229c67128" containerName="istio-proxy" Nov 04 04:04:13.738574 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738507 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388ae503-75e8-4507-b6c2-9e4229c67128" containerName="istio-proxy" Nov 04 04:04:13.738574 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738522 4966 cpu_manager.go:414] "containerMap: removing container" podUID="28e01b06-4f41-4ec9-b351-6145f5d0bd2a" containerName="payload-pre-processing" Nov 04 04:04:13.738574 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738529 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e01b06-4f41-4ec9-b351-6145f5d0bd2a" containerName="payload-pre-processing" Nov 04 04:04:13.738574 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738543 4966 cpu_manager.go:414] "containerMap: removing container" podUID="6626886d-db18-4cd8-ae04-23e2b5841223" containerName="payload-processing" Nov 04 04:04:13.738574 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738550 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6626886d-db18-4cd8-ae04-23e2b5841223" containerName="payload-processing" Nov 04 04:04:13.738574 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738563 4966 cpu_manager.go:414] "containerMap: removing container" podUID="2060984f-5a85-4790-b19c-3e6fff9016ae" containerName="istio-proxy" Nov 04 04:04:13.738574 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738570 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2060984f-5a85-4790-b19c-3e6fff9016ae" containerName="istio-proxy" Nov 04 04:04:13.738864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738592 4966 cpu_manager.go:414] "containerMap: removing container" podUID="c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1" containerName="payload-pre-processing" Nov 04 04:04:13.738864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738599 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1" containerName="payload-pre-processing" Nov 04 04:04:13.738864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738612 4966 cpu_manager.go:414] "containerMap: removing container" podUID="76911349-4357-4a41-9218-a6077aed29bc" containerName="main" Nov 04 04:04:13.738864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738619 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76911349-4357-4a41-9218-a6077aed29bc" containerName="main" Nov 04 04:04:13.738864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738633 4966 cpu_manager.go:414] "containerMap: removing container" podUID="8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" containerName="istio-proxy" Nov 04 04:04:13.738864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738640 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" containerName="istio-proxy" Nov 04 04:04:13.738864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738741 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7920bcb-2a60-45c0-bc3d-0d28ae3fc3d1" containerName="payload-pre-processing" Nov 04 04:04:13.738864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738752 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="76911349-4357-4a41-9218-a6077aed29bc" containerName="main" Nov 04 04:04:13.738864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738761 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d1303a0-cb6b-4d05-9b03-bc3eb0e62bd4" containerName="istio-proxy" Nov 04 04:04:13.738864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738775 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0e1ba5-30ca-49f3-be4b-2e5a474d036a" containerName="istio-proxy" Nov 04 04:04:13.738864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738784 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff721376-eba2-4a06-b1b9-48e5875d9c05" containerName="maas-api" Nov 04 04:04:13.738864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738793 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed1dacd-75d6-4e61-94bb-d7fcced94f85" containerName="istio-proxy" Nov 04 04:04:13.738864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738799 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="388ae503-75e8-4507-b6c2-9e4229c67128" containerName="istio-proxy" Nov 04 04:04:13.738864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738808 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e01b06-4f41-4ec9-b351-6145f5d0bd2a" containerName="payload-pre-processing" Nov 04 04:04:13.738864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738817 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6626886d-db18-4cd8-ae04-23e2b5841223" containerName="payload-processing" Nov 04 04:04:13.738864 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.738825 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="2060984f-5a85-4790-b19c-3e6fff9016ae" containerName="istio-proxy" Nov 04 04:04:13.742563 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.742542 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" Nov 04 04:04:13.749906 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.749840 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b"] Nov 04 04:04:13.792577 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.792547 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26w7b\"/\"kube-root-ca.crt\"" Nov 04 04:04:13.792728 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.792635 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26w7b\"/\"openshift-service-ca.crt\"" Nov 04 04:04:13.804284 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.804253 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/590c3c1c-ec8b-4c7c-9e87-568790bd9c35-sys\") pod \"perf-node-gather-daemonset-8hs8b\" (UID: \"590c3c1c-ec8b-4c7c-9e87-568790bd9c35\") " pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" Nov 04 04:04:13.804398 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.804295 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/590c3c1c-ec8b-4c7c-9e87-568790bd9c35-lib-modules\") pod \"perf-node-gather-daemonset-8hs8b\" (UID: \"590c3c1c-ec8b-4c7c-9e87-568790bd9c35\") " pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" Nov 04 04:04:13.804398 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.804335 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/590c3c1c-ec8b-4c7c-9e87-568790bd9c35-proc\") pod \"perf-node-gather-daemonset-8hs8b\" (UID: \"590c3c1c-ec8b-4c7c-9e87-568790bd9c35\") " pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" Nov 04 04:04:13.804480 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.804459 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/590c3c1c-ec8b-4c7c-9e87-568790bd9c35-podres\") pod \"perf-node-gather-daemonset-8hs8b\" (UID: \"590c3c1c-ec8b-4c7c-9e87-568790bd9c35\") " pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" Nov 04 04:04:13.804517 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.804498 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt9jv\" (UniqueName: \"kubernetes.io/projected/590c3c1c-ec8b-4c7c-9e87-568790bd9c35-kube-api-access-kt9jv\") pod \"perf-node-gather-daemonset-8hs8b\" (UID: \"590c3c1c-ec8b-4c7c-9e87-568790bd9c35\") " pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" Nov 04 04:04:13.834963 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.834914 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-26w7b\"/\"default-dockercfg-4hjnm\"" Nov 04 04:04:13.905658 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.905625 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/590c3c1c-ec8b-4c7c-9e87-568790bd9c35-podres\") pod \"perf-node-gather-daemonset-8hs8b\" (UID: \"590c3c1c-ec8b-4c7c-9e87-568790bd9c35\") " pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" Nov 04 04:04:13.905814 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.905674 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt9jv\" (UniqueName: \"kubernetes.io/projected/590c3c1c-ec8b-4c7c-9e87-568790bd9c35-kube-api-access-kt9jv\") pod \"perf-node-gather-daemonset-8hs8b\" (UID: \"590c3c1c-ec8b-4c7c-9e87-568790bd9c35\") " pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" Nov 04 04:04:13.905814 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.905717 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/590c3c1c-ec8b-4c7c-9e87-568790bd9c35-podres\") pod \"perf-node-gather-daemonset-8hs8b\" (UID: \"590c3c1c-ec8b-4c7c-9e87-568790bd9c35\") " pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" Nov 04 04:04:13.905814 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.905781 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/590c3c1c-ec8b-4c7c-9e87-568790bd9c35-sys\") pod \"perf-node-gather-daemonset-8hs8b\" (UID: \"590c3c1c-ec8b-4c7c-9e87-568790bd9c35\") " pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" Nov 04 04:04:13.905814 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.905811 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/590c3c1c-ec8b-4c7c-9e87-568790bd9c35-lib-modules\") pod \"perf-node-gather-daemonset-8hs8b\" (UID: \"590c3c1c-ec8b-4c7c-9e87-568790bd9c35\") " pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" Nov 04 04:04:13.905988 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.905857 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/590c3c1c-ec8b-4c7c-9e87-568790bd9c35-proc\") pod \"perf-node-gather-daemonset-8hs8b\" (UID: \"590c3c1c-ec8b-4c7c-9e87-568790bd9c35\") " pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" Nov 04 04:04:13.905988 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.905862 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/590c3c1c-ec8b-4c7c-9e87-568790bd9c35-sys\") pod \"perf-node-gather-daemonset-8hs8b\" (UID: \"590c3c1c-ec8b-4c7c-9e87-568790bd9c35\") " pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" Nov 04 04:04:13.905988 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.905912 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/590c3c1c-ec8b-4c7c-9e87-568790bd9c35-proc\") pod \"perf-node-gather-daemonset-8hs8b\" (UID: \"590c3c1c-ec8b-4c7c-9e87-568790bd9c35\") " pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" Nov 04 04:04:13.905988 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.905958 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/590c3c1c-ec8b-4c7c-9e87-568790bd9c35-lib-modules\") pod \"perf-node-gather-daemonset-8hs8b\" (UID: \"590c3c1c-ec8b-4c7c-9e87-568790bd9c35\") " pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" Nov 04 04:04:13.918511 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:13.918480 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt9jv\" (UniqueName: \"kubernetes.io/projected/590c3c1c-ec8b-4c7c-9e87-568790bd9c35-kube-api-access-kt9jv\") pod \"perf-node-gather-daemonset-8hs8b\" (UID: \"590c3c1c-ec8b-4c7c-9e87-568790bd9c35\") " pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" Nov 04 04:04:14.058072 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:14.058037 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" Nov 04 04:04:14.337881 ip-10-0-138-9 kubenswrapper[4966]: W1104 04:04:14.337851 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod590c3c1c_ec8b_4c7c_9e87_568790bd9c35.slice/crio-ffd056e9413678abfb19e81c37de289a6c167fb2250d71a2cf1ae0fcb946489f WatchSource:0}: Error finding container ffd056e9413678abfb19e81c37de289a6c167fb2250d71a2cf1ae0fcb946489f: Status 404 returned error can't find the container with id ffd056e9413678abfb19e81c37de289a6c167fb2250d71a2cf1ae0fcb946489f Nov 04 04:04:14.343020 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:14.342996 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b"] Nov 04 04:04:14.497075 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:14.497040 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" event={"ID":"590c3c1c-ec8b-4c7c-9e87-568790bd9c35","Type":"ContainerStarted","Data":"edc08da8df9d94042f869b5a2715c8c85999c5368d2a76dae68ef2797949e8ee"} Nov 04 04:04:14.497205 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:14.497097 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" event={"ID":"590c3c1c-ec8b-4c7c-9e87-568790bd9c35","Type":"ContainerStarted","Data":"ffd056e9413678abfb19e81c37de289a6c167fb2250d71a2cf1ae0fcb946489f"} Nov 04 04:04:14.497264 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:14.497248 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" Nov 04 04:04:14.517837 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:14.517792 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" podStartSLOduration=1.5177795280000002 podStartE2EDuration="1.517779528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:04:14.51520399 +0000 UTC m=+898.316149798" watchObservedRunningTime="2026-11-04 04:04:14.517779528 +0000 UTC m=+898.318725336" Nov 04 04:04:14.994759 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:14.994727 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-2hhp2_f37b227d-fc9c-4a81-85b1-62c04243ad83/networking-console-plugin/0.log" Nov 04 04:04:15.550999 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:15.550967 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/3.log" Nov 04 04:04:15.554828 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:15.554803 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/4.log" Nov 04 04:04:16.484587 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:16.484553 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/3.log" Nov 04 04:04:16.488166 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:16.488142 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rr5bv_9f66afa6-a9ae-4e9b-84e4-b2fbf4136da6/console-operator/3.log" Nov 04 04:04:16.716870 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:16.716841 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-vjhfk_3fbef9e3-d5de-4a35-8745-2b1dc357c52b/volume-data-source-validator/0.log" Nov 04 04:04:18.078580 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:18.078542 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pvrd4_7b632ca9-a695-450e-a1a9-b5357ebf4d7d/dns/0.log" Nov 04 04:04:18.106484 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:18.106454 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pvrd4_7b632ca9-a695-450e-a1a9-b5357ebf4d7d/kube-rbac-proxy/0.log" Nov 04 04:04:18.226873 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:18.226845 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c8xnp_c12382e2-90c4-45bf-bdd8-27653073cc5d/dns-node-resolver/0.log" Nov 04 04:04:18.880305 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:18.880271 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5786847478-jw69k_e4cc7333-a66b-42e0-a925-7a3be02d606e/registry/0.log" Nov 04 04:04:18.970750 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:18.970722 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jkpjj_795d3139-8d19-4852-bd9c-3c60b01a97f6/node-ca/0.log" Nov 04 04:04:20.081938 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:20.081887 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5ck24_9747b332-91da-4e7c-ae71-700ac67567f9/discovery/0.log" Nov 04 04:04:20.515478 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:20.515393 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-26w7b/perf-node-gather-daemonset-8hs8b" Nov 04 04:04:20.583777 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:20.583750 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bd469f65f-x7t8v_f376642e-e719-4112-858d-5dc9575b1236/kube-auth-proxy/0.log" Nov 04 04:04:20.617547 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:20.617517 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-4tpt8_177f8aff-8b11-4891-9648-9c7aeb9305b7/istio-proxy/0.log" Nov 04 04:04:20.729517 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:20.729464 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-n5x92_aa513df7-058f-4dbc-8743-f0b1e3f5e088/istio-proxy/0.log" Nov 04 04:04:20.862354 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:20.862326 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf_12af8c07-513e-4b8c-8172-4aebb9731f66/istio-proxy/0.log" Nov 04 04:04:20.892470 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:20.892442 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-lqlbm_c5ea3aa4-a31f-430e-ab69-31d81f975218/payload-pre-processing/0.log" Nov 04 04:04:20.932063 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:20.932037 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-sv8xh_94562f01-24b5-4df4-ae94-967d5a116a9c/payload-processing/0.log" Nov 04 04:04:20.967011 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:20.966987 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78ccf8f879-s4sx4_1b81b9bc-4eb5-48a7-98e1-6c7a98117f57/router/0.log" Nov 04 04:04:22.062372 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:22.062342 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-chk4t_bcc0d689-48b6-467e-bdd0-6a03878a52b6/serve-healthcheck-canary/0.log" Nov 04 04:04:22.784115 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:22.784053 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-gm5s7_3b456f4f-071a-4e5f-b8b6-51c3ee790656/insights-operator/1.log" Nov 04 04:04:22.784458 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:22.784440 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-gm5s7_3b456f4f-071a-4e5f-b8b6-51c3ee790656/insights-operator/2.log" Nov 04 04:04:22.999090 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:22.999062 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zfjhp_78e55dc9-354d-455f-9d02-b3cfd5885d65/kube-rbac-proxy/0.log" Nov 04 04:04:23.039702 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:23.039605 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zfjhp_78e55dc9-354d-455f-9d02-b3cfd5885d65/exporter/0.log" Nov 04 04:04:23.071733 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:23.071690 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zfjhp_78e55dc9-354d-455f-9d02-b3cfd5885d65/extractor/0.log" Nov 04 04:04:24.944015 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:24.943960 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5ck24_9747b332-91da-4e7c-ae71-700ac67567f9/discovery/0.log" Nov 04 04:04:25.423533 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:25.423474 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bd469f65f-x7t8v_f376642e-e719-4112-858d-5dc9575b1236/kube-auth-proxy/0.log" Nov 04 04:04:25.445859 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:25.445831 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-4tpt8_177f8aff-8b11-4891-9648-9c7aeb9305b7/istio-proxy/0.log" Nov 04 04:04:25.509845 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:25.509808 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-n5x92_aa513df7-058f-4dbc-8743-f0b1e3f5e088/istio-proxy/0.log" Nov 04 04:04:25.613534 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:25.613509 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf_12af8c07-513e-4b8c-8172-4aebb9731f66/istio-proxy/0.log" Nov 04 04:04:25.631772 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:25.631747 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-lqlbm_c5ea3aa4-a31f-430e-ab69-31d81f975218/payload-pre-processing/0.log" Nov 04 04:04:25.649147 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:25.649124 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-sv8xh_94562f01-24b5-4df4-ae94-967d5a116a9c/payload-processing/0.log" Nov 04 04:04:25.674875 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:25.674812 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78ccf8f879-s4sx4_1b81b9bc-4eb5-48a7-98e1-6c7a98117f57/router/0.log" Nov 04 04:04:26.183672 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:26.183645 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-74f9c8978b-zkjxq_147eeb22-995f-45be-908b-8a6edce063b1/manager/0.log" Nov 04 04:04:26.204895 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:26.204870 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-74f9c8978b-zkjxq_147eeb22-995f-45be-908b-8a6edce063b1/copy-manifests/0.log" Nov 04 04:04:27.208766 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:27.208737 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5ck24_9747b332-91da-4e7c-ae71-700ac67567f9/discovery/0.log" Nov 04 04:04:27.418000 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:27.417953 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-b659x_3caf95e8-e1ce-488c-aa79-9d558ad1caed/jobset-operator/0.log" Nov 04 04:04:27.693147 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:27.693117 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bd469f65f-x7t8v_f376642e-e719-4112-858d-5dc9575b1236/kube-auth-proxy/0.log" Nov 04 04:04:27.712753 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:27.712728 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-4tpt8_177f8aff-8b11-4891-9648-9c7aeb9305b7/istio-proxy/0.log" Nov 04 04:04:27.787678 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:27.787651 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-n5x92_aa513df7-058f-4dbc-8743-f0b1e3f5e088/istio-proxy/0.log" Nov 04 04:04:27.868671 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:27.868643 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf_12af8c07-513e-4b8c-8172-4aebb9731f66/istio-proxy/0.log" Nov 04 04:04:27.886879 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:27.886837 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-lqlbm_c5ea3aa4-a31f-430e-ab69-31d81f975218/payload-pre-processing/0.log" Nov 04 04:04:27.903550 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:27.903529 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-sv8xh_94562f01-24b5-4df4-ae94-967d5a116a9c/payload-processing/0.log" Nov 04 04:04:27.928177 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:27.928154 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78ccf8f879-s4sx4_1b81b9bc-4eb5-48a7-98e1-6c7a98117f57/router/0.log" Nov 04 04:04:28.087216 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:28.087186 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-58vj8_545fb734-bf40-43f8-a19d-0fe82840ef71/manager/0.log" Nov 04 04:04:28.124131 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:28.124088 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sdgx5_f2f40e58-66ab-4754-bd77-ed8e3d20a818/manager/0.log" Nov 04 04:04:28.440007 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:28.439869 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55cc4d9668-f6wtl_c4c6cd4f-a5bc-4602-b6cc-e8ed6595ed59/manager/0.log" Nov 04 04:04:28.463459 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:28.463422 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-2m46k_bffcd9a5-5579-4223-ad60-f1c5a862645f/server/0.log" Nov 04 04:04:28.651114 ip-10-0-138-9 kubenswrapper[4966]: I1104 04:04:28.651082 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-xdp9r_06eeac89-43b5-4e21-bd3c-e480330635e8/manager/0.log"