Jul 03 14:06:30.781652 ip-10-0-140-82 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 03 14:06:30.781666 ip-10-0-140-82 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 03 14:06:30.781675 ip-10-0-140-82 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 03 14:06:30.781980 ip-10-0-140-82 systemd[1]: Failed to start Kubernetes Kubelet. Jul 03 14:06:40.925322 ip-10-0-140-82 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 03 14:06:40.925340 ip-10-0-140-82 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 46bf28d57a754e6e884bd931353e3b31 -- Jul 03 14:08:40.484570 ip-10-0-140-82 systemd[1]: Starting Kubernetes Kubelet... Jul 03 14:08:40.914463 ip-10-0-140-82 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 03 14:08:40.914463 ip-10-0-140-82 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 03 14:08:40.914463 ip-10-0-140-82 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 03 14:08:40.914463 ip-10-0-140-82 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 03 14:08:40.914463 ip-10-0-140-82 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 03 14:08:40.917327 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.917224 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 03 14:08:40.924613 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924589 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 14:08:40.924613 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924608 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 14:08:40.924613 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924612 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 14:08:40.924613 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924616 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 14:08:40.924613 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924619 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 14:08:40.924613 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924621 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924624 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924627 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924630 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924634 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924637 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924639 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924642 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924645 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924647 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924650 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924652 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924655 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924658 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924661 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924663 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924665 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924668 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924670 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924673 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 14:08:40.924835 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924676 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 14:08:40.925336 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924678 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 14:08:40.925336 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924682 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 14:08:40.925336 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924684 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 14:08:40.925336 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924687 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 14:08:40.925336 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924690 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 14:08:40.925336 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924718 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 14:08:40.925336 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924722 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 14:08:40.925336 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924725 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 14:08:40.925336 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924728 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 14:08:40.925336 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924731 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 14:08:40.925336 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924734 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 14:08:40.925336 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924737 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 14:08:40.925336 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924740 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 14:08:40.925336 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924743 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 14:08:40.925336 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924746 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 14:08:40.925336 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924749 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 14:08:40.925336 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924751 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 14:08:40.925336 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924754 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 14:08:40.925336 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924758 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924763 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924766 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924769 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924772 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924775 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924778 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924781 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924784 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924786 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924789 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924792 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924795 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924797 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924800 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924804 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924808 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924811 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924813 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924816 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 14:08:40.925805 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924819 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924821 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924824 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924826 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924828 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924831 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924833 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924836 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924839 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924842 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924845 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924848 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924852 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924856 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924859 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924862 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924866 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924868 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924872 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924875 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 14:08:40.926303 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924878 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 14:08:40.926778 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924881 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 14:08:40.926778 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924884 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 14:08:40.926778 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924887 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 14:08:40.926778 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924889 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 14:08:40.926778 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924892 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 14:08:40.926778 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924894 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 14:08:40.926778 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924896 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 14:08:40.926778 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924899 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 14:08:40.926778 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924902 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 14:08:40.926778 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924905 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 14:08:40.926778 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924908 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 14:08:40.926778 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924911 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 14:08:40.926778 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924914 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 14:08:40.926778 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.924916 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 14:08:40.926778 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926660 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 14:08:40.926778 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926668 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 14:08:40.926778 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926671 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 14:08:40.926778 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926674 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 14:08:40.926778 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926677 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926680 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926683 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926686 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926688 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926691 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926693 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926695 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926698 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926700 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926703 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926706 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926709 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926712 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926714 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926717 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926720 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926723 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926725 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926728 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 14:08:40.927246 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926730 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926733 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926735 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926738 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926740 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926743 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926745 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926748 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926750 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926752 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926755 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926757 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926760 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926762 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926765 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926767 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926769 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926773 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926775 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926778 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 14:08:40.927754 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926780 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 14:08:40.928240 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926783 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 14:08:40.928240 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926786 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 14:08:40.928240 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926788 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 14:08:40.928240 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926792 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 14:08:40.928240 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926794 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 14:08:40.928240 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926797 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 14:08:40.928240 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926800 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 14:08:40.928240 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926803 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 14:08:40.928240 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926806 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 14:08:40.928240 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926808 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 14:08:40.928240 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926810 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 14:08:40.928240 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926813 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 14:08:40.928240 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926815 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 14:08:40.928240 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926818 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 14:08:40.928240 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926820 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 14:08:40.928240 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926822 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 14:08:40.928240 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926824 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 14:08:40.928240 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926827 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 14:08:40.928240 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926831 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 14:08:40.928728 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926834 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 14:08:40.928728 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926837 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 14:08:40.928728 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926840 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 14:08:40.928728 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926843 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 14:08:40.928728 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926845 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 14:08:40.928728 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926848 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 14:08:40.928728 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926851 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 14:08:40.928728 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926853 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 14:08:40.928728 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926856 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 14:08:40.928728 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926858 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 14:08:40.928728 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926860 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 14:08:40.928728 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926863 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 14:08:40.928728 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926865 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 14:08:40.928728 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926870 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 14:08:40.928728 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926874 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 14:08:40.928728 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926877 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 14:08:40.928728 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926880 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 14:08:40.928728 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926882 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 14:08:40.928728 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926885 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 14:08:40.929197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926891 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 14:08:40.929197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926894 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 14:08:40.929197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926897 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 14:08:40.929197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926899 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 14:08:40.929197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926903 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 14:08:40.929197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926907 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 14:08:40.929197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926910 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 14:08:40.929197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926912 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 14:08:40.929197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926915 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 14:08:40.929197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926917 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 14:08:40.929197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926920 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 14:08:40.929197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926922 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 14:08:40.929197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926924 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 14:08:40.929197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926927 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 14:08:40.929197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926929 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 14:08:40.929197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.926932 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 14:08:40.929197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927382 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 14:08:40.929197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927388 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 03 14:08:40.929197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927392 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927395 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927398 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927401 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927404 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927408 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927413 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927417 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927421 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927424 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927429 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927433 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927437 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927440 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927446 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927448 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927451 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927454 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927457 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927459 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 14:08:40.929692 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927462 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927466 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927470 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927473 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927475 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927478 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927481 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927484 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927486 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927489 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927491 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927494 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927496 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927498 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927501 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927503 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927506 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927508 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927511 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927513 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 14:08:40.930173 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927516 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927518 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927521 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927523 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927526 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927529 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927533 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927537 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927539 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927542 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927544 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927547 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927549 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927552 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927555 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927557 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927560 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927562 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927565 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927567 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 14:08:40.930664 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927569 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 14:08:40.931192 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927572 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 14:08:40.931192 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927574 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 14:08:40.931192 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927577 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 14:08:40.931192 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927579 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 14:08:40.931192 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927582 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 14:08:40.931192 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927584 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 14:08:40.931192 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927587 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 14:08:40.931192 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927589 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 14:08:40.931192 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927592 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 14:08:40.931192 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927594 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 14:08:40.931192 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927596 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 14:08:40.931192 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927599 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 14:08:40.931192 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927601 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 14:08:40.931192 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927604 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 14:08:40.931192 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927606 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 14:08:40.931192 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927609 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 14:08:40.931192 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927612 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 14:08:40.931192 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927615 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 14:08:40.931192 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927618 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927621 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927623 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927626 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927628 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927631 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927633 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927636 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927640 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927642 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927646 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927650 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927654 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927657 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927660 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927662 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927665 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.927668 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927734 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927749 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 03 14:08:40.931670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927755 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927762 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927767 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927770 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927774 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927778 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927782 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927784 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927788 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927791 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927794 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927798 2576 flags.go:64] FLAG: --cgroup-root="" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927801 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927804 2576 flags.go:64] FLAG: --client-ca-file="" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927807 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927810 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927815 2576 flags.go:64] FLAG: --cluster-domain="" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927818 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927821 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927824 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927829 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927833 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927836 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 03 14:08:40.932152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927839 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927842 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927845 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927848 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927851 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927854 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927857 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927861 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927864 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927867 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927870 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927872 2576 flags.go:64] FLAG: --enable-server="true" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927875 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927879 2576 flags.go:64] FLAG: --event-burst="100" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927882 2576 flags.go:64] FLAG: --event-qps="50" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927885 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927888 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927891 2576 flags.go:64] FLAG: --eviction-hard="" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927894 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927897 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927902 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927905 2576 flags.go:64] FLAG: --eviction-soft="" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927908 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927910 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927915 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 03 14:08:40.932749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927918 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927920 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927923 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927926 2576 flags.go:64] FLAG: --feature-gates="" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927931 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927934 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927937 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927940 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927943 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927946 2576 flags.go:64] FLAG: --help="false" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927948 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-140-82.ec2.internal" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927951 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927954 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927957 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927960 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927963 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927966 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927969 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927972 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927974 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927977 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927980 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927983 2576 flags.go:64] FLAG: --kube-reserved="" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927985 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 03 14:08:40.933458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927988 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927991 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927993 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.927997 2576 flags.go:64] FLAG: --lock-file="" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928000 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928003 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928006 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928011 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928014 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928017 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928020 2576 flags.go:64] FLAG: --logging-format="text" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928022 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928027 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928030 2576 flags.go:64] FLAG: --manifest-url="" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928033 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928037 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928040 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928044 2576 flags.go:64] FLAG: --max-pods="110" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928047 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928050 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928052 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928055 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928058 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928060 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928063 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 03 14:08:40.934039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928071 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928073 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928076 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928079 2576 flags.go:64] FLAG: --pod-cidr="" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928082 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928088 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928091 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928094 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928097 2576 flags.go:64] FLAG: --port="10250" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928100 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928103 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06ea631260186bc68" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928107 2576 flags.go:64] FLAG: --qos-reserved="" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928110 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928113 2576 flags.go:64] FLAG: --register-node="true" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928116 2576 flags.go:64] FLAG: --register-with-taints="" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928119 2576 flags.go:64] FLAG: --registry-burst="10" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928123 2576 flags.go:64] FLAG: --registry-qps="5" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928125 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928128 2576 flags.go:64] FLAG: --reserved-memory="" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928133 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928136 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928139 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928141 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928144 2576 flags.go:64] FLAG: --runonce="false" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928147 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 03 14:08:40.934670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928150 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928152 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928155 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928158 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928161 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928164 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928167 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928170 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928172 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928175 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928178 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928181 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928183 2576 flags.go:64] FLAG: --system-cgroups="" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928186 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928191 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928194 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928196 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928203 2576 flags.go:64] FLAG: --tls-min-version="" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928207 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928210 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928213 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928215 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928218 2576 flags.go:64] FLAG: --v="2" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928222 2576 flags.go:64] FLAG: --version="false" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928226 2576 flags.go:64] FLAG: --vmodule="" Jul 03 14:08:40.935340 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928230 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.928235 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928384 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928389 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928392 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928396 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928399 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928401 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928405 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928407 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928410 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928413 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928415 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928418 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928420 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928423 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928425 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928428 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928431 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928433 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 14:08:40.935961 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928435 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 14:08:40.936515 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928438 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 14:08:40.936515 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928440 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 14:08:40.936515 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928442 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 14:08:40.936515 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928445 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 14:08:40.936515 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928448 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 14:08:40.936515 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928451 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 14:08:40.936515 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928453 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 14:08:40.936515 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928456 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 14:08:40.936515 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928458 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 14:08:40.936515 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928461 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 14:08:40.936515 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928463 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 14:08:40.936515 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928466 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 14:08:40.936515 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928470 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 14:08:40.936515 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928473 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 14:08:40.936515 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928478 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 14:08:40.936515 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928481 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 14:08:40.936515 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928484 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 14:08:40.936515 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928487 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 14:08:40.936515 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928490 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928492 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928495 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928497 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928500 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928502 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928504 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928507 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928510 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928512 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928515 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928517 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928520 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928522 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928525 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928527 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928529 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928532 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928534 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928537 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 14:08:40.936978 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928539 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 14:08:40.937590 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928542 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 14:08:40.937590 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928544 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 14:08:40.937590 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928547 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 14:08:40.937590 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928551 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 14:08:40.937590 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928555 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 14:08:40.937590 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928561 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 14:08:40.937590 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928564 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 14:08:40.937590 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928567 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 14:08:40.937590 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928570 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 14:08:40.937590 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928573 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 14:08:40.937590 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928575 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 14:08:40.937590 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928578 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 14:08:40.937590 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928580 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 14:08:40.937590 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928583 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 14:08:40.937590 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928585 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 14:08:40.937590 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928588 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 14:08:40.937590 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928590 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 14:08:40.937590 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928592 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 14:08:40.937590 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928595 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928597 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928599 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928602 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928604 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928607 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928609 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928612 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928614 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928616 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928619 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928621 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928624 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928627 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928629 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928632 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928634 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928637 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928639 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928643 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 14:08:40.938055 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928646 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 14:08:40.938555 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928649 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 14:08:40.938555 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.928651 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 14:08:40.938555 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.929280 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 03 14:08:40.938555 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.936460 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 03 14:08:40.938555 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.936477 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 03 14:08:40.938555 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.936498 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 03 14:08:40.938555 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.936504 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 03 14:08:40.938555 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936574 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 14:08:40.938555 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936579 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 14:08:40.938555 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936582 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 14:08:40.938555 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936585 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 14:08:40.938555 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936588 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 14:08:40.938555 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936591 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 14:08:40.938555 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936594 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 14:08:40.938555 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936596 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936599 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936602 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936604 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936607 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936609 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936612 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936614 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936617 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936620 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936622 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936625 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936627 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936630 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936633 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936635 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936638 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936640 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936643 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936645 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 14:08:40.938927 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936647 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 14:08:40.939434 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936650 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 14:08:40.939434 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936654 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 14:08:40.939434 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936657 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 14:08:40.939434 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936660 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 14:08:40.939434 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936662 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 14:08:40.939434 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936665 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 14:08:40.939434 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936667 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 14:08:40.939434 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936669 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 14:08:40.939434 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936673 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 14:08:40.939434 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936677 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 14:08:40.939434 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936680 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 14:08:40.939434 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936682 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 14:08:40.939434 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936685 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 14:08:40.939434 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936687 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 14:08:40.939434 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936690 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 14:08:40.939434 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936692 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 14:08:40.939434 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936695 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 03 14:08:40.939434 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936697 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 14:08:40.939434 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936701 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 14:08:40.939929 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936703 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 14:08:40.939929 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936706 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 14:08:40.939929 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936708 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 14:08:40.939929 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936711 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 14:08:40.939929 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936713 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 14:08:40.939929 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936716 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 14:08:40.939929 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936718 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 14:08:40.939929 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936721 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 14:08:40.939929 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936723 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 14:08:40.939929 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936727 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 14:08:40.939929 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936731 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 14:08:40.939929 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936734 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 14:08:40.939929 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936737 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 14:08:40.939929 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936739 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 14:08:40.939929 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936743 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 14:08:40.939929 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936746 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 14:08:40.939929 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936750 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 14:08:40.939929 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936752 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 14:08:40.939929 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936756 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936759 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936762 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936765 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936768 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936770 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936773 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936775 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936778 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936780 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936782 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936785 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936787 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936789 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936792 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936795 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936797 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936799 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936802 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936804 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 14:08:40.940418 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936807 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 14:08:40.940898 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936809 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 14:08:40.940898 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936811 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 14:08:40.940898 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936814 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 14:08:40.940898 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936816 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 14:08:40.940898 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936819 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 14:08:40.940898 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936821 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 14:08:40.940898 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936824 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 14:08:40.940898 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936827 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 14:08:40.940898 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936830 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 14:08:40.940898 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936836 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 14:08:40.940898 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936838 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 14:08:40.940898 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936841 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 14:08:40.940898 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936843 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 14:08:40.940898 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.936848 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 03 14:08:40.940898 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936967 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 14:08:40.941397 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936973 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 14:08:40.941397 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936976 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 14:08:40.941397 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936979 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 14:08:40.941397 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936981 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 14:08:40.941397 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936984 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 14:08:40.941397 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936987 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 14:08:40.941397 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936990 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 14:08:40.941397 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936994 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 14:08:40.941397 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.936997 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 14:08:40.941397 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937001 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 14:08:40.941397 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937004 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 14:08:40.941397 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937007 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 14:08:40.941397 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937010 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 14:08:40.941397 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937012 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 14:08:40.941397 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937015 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 14:08:40.941397 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937018 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 14:08:40.941397 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937021 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 14:08:40.941397 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937023 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 14:08:40.941397 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937026 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937029 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937031 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937034 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937036 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937039 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937042 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937045 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937048 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937051 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937054 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937056 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937059 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937061 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937064 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937066 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937068 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937071 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937073 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937076 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 14:08:40.941868 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937078 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 14:08:40.942464 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937081 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 14:08:40.942464 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937083 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 14:08:40.942464 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937086 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 14:08:40.942464 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937088 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 14:08:40.942464 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937091 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 14:08:40.942464 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937093 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 14:08:40.942464 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937096 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 14:08:40.942464 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937098 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 14:08:40.942464 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937100 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 14:08:40.942464 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937103 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 14:08:40.942464 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937105 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 14:08:40.942464 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937108 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 14:08:40.942464 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937110 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 14:08:40.942464 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937112 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 14:08:40.942464 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937115 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 14:08:40.942464 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937117 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 14:08:40.942464 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937120 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 14:08:40.942464 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937123 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 14:08:40.942464 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937126 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 14:08:40.942984 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937129 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 14:08:40.942984 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937133 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 14:08:40.942984 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937136 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 14:08:40.942984 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937139 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 14:08:40.942984 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937141 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 14:08:40.942984 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937144 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 14:08:40.942984 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937147 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 14:08:40.942984 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937150 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 14:08:40.942984 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937152 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 14:08:40.942984 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937155 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 14:08:40.942984 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937157 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 14:08:40.942984 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937160 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 14:08:40.942984 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937162 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 14:08:40.942984 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937164 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 14:08:40.942984 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937167 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 14:08:40.942984 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937169 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 14:08:40.942984 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937172 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 14:08:40.942984 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937174 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 14:08:40.942984 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937176 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937179 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937182 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937184 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937186 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937189 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937191 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937194 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937196 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937198 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937201 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937203 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937206 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937208 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937211 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937213 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937216 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937219 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937222 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937224 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 14:08:40.943785 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937227 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 14:08:40.944408 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:40.937230 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 14:08:40.944408 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.937234 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 03 14:08:40.944408 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.937377 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 03 14:08:40.944408 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.939440 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 03 14:08:40.944408 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.940924 2576 server.go:1019] "Starting client certificate rotation" Jul 03 14:08:40.944408 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.941056 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 03 14:08:40.944408 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.941111 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 03 14:08:40.944408 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.943437 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 03 14:08:40.944725 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.944696 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 03 14:08:40.953332 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.953303 2576 log.go:25] "Validated CRI v1 runtime API" Jul 03 14:08:40.955437 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.955422 2576 log.go:25] "Validated CRI v1 image API" Jul 03 14:08:40.956893 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.956877 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 03 14:08:40.960186 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.960162 2576 fs.go:135] Filesystem UUIDs: map[6926692c-c1ed-4428-af2d-16327a89e5d5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8679dfd3-bf79-4be2-a5e0-1c26cce3ffaa:/dev/nvme0n1p3] Jul 03 14:08:40.960300 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.960183 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 03 14:08:40.965321 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.965179 2576 manager.go:216] Machine: {Timestamp:2026-07-03 14:08:40.963815448 +0000 UTC m=+0.355493168 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3147769 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23c649a5920656dfc74f1e409c295d SystemUUID:ec23c649-a592-0656-dfc7-4f1e409c295d BootID:46bf28d5-7a75-4e6e-884b-d931353e3b31 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:12:b7:a3:e9:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:12:b7:a3:e9:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:e2:19:ad:70:07 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 03 14:08:40.965321 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.965311 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 03 14:08:40.965439 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.965401 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 03 14:08:40.966620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.966590 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 03 14:08:40.966770 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.966623 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-82.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 03 14:08:40.967561 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.967551 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 03 14:08:40.967594 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.967563 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 03 14:08:40.967594 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.967570 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 03 14:08:40.968194 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.968183 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 03 14:08:40.969593 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.969581 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 03 14:08:40.972922 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.972908 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 03 14:08:40.972986 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.972973 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 03 14:08:40.973098 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.973088 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 03 14:08:40.973133 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.973124 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 03 14:08:40.973166 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.973141 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 03 14:08:40.973166 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.973154 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 03 14:08:40.973219 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.973185 2576 kubelet.go:398] "Adding apiserver pod source" Jul 03 14:08:40.973219 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.973196 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 03 14:08:40.973620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.973609 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 03 14:08:40.973673 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.973627 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 03 14:08:40.975839 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.975822 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 03 14:08:40.977882 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.977868 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 03 14:08:40.977938 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.977899 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 03 14:08:40.979355 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.979344 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 03 14:08:40.979400 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.979372 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 03 14:08:40.979400 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.979378 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 03 14:08:40.979400 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.979388 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 03 14:08:40.979400 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.979396 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 03 14:08:40.979400 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.979402 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 03 14:08:40.979529 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.979410 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 03 14:08:40.979529 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.979415 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 03 14:08:40.979529 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.979421 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 03 14:08:40.979529 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.979426 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 03 14:08:40.979529 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.979437 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 03 14:08:40.979529 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.979442 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 03 14:08:40.980133 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.980115 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 03 14:08:40.980133 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.980126 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 03 14:08:40.984302 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.984283 2576 server.go:1268] "Started kubelet" Jul 03 14:08:40.984564 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.984489 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 03 14:08:40.984637 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.984572 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 03 14:08:40.985129 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.985097 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 03 14:08:40.985138 ip-10-0-140-82 systemd[1]: Started Kubernetes Kubelet. Jul 03 14:08:40.985797 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.985784 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 03 14:08:40.986378 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:40.986346 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 03 14:08:40.986461 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:40.986424 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 03 14:08:40.986530 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.986507 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 03 14:08:40.986768 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.986752 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-82.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 03 14:08:40.989147 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:40.989110 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 03 14:08:40.991034 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:40.989979 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-82.ec2.internal.18beccd5c49fa35d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-82.ec2.internal,UID:ip-10-0-140-82.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-82.ec2.internal,},FirstTimestamp:2026-07-03 14:08:40.984191837 +0000 UTC m=+0.375869556,LastTimestamp:2026-07-03 14:08:40.984191837 +0000 UTC m=+0.375869556,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-82.ec2.internal,}" Jul 03 14:08:40.991696 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.991683 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 03 14:08:40.991754 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.991695 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 03 14:08:40.991818 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:40.991752 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-82.ec2.internal\" not found" Jul 03 14:08:40.992365 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.992345 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 03 14:08:40.992365 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.992365 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 03 14:08:40.992510 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.992345 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 03 14:08:40.992585 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.992571 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 03 14:08:40.992708 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.992634 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 03 14:08:40.992811 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.992788 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 03 14:08:40.992864 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.992824 2576 factory.go:55] Registering systemd factory Jul 03 14:08:40.992864 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.992834 2576 factory.go:223] Registration of the systemd container factory successfully Jul 03 14:08:40.993126 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.993102 2576 factory.go:153] Registering CRI-O factory Jul 03 14:08:40.993126 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.993121 2576 factory.go:223] Registration of the crio container factory successfully Jul 03 14:08:40.993292 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.993159 2576 factory.go:103] Registering Raw factory Jul 03 14:08:40.993292 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.993181 2576 manager.go:1196] Started watching for new ooms in manager Jul 03 14:08:40.993576 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:40.993563 2576 manager.go:349] Starting recovery of all containers Jul 03 14:08:40.998638 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:40.998613 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 03 14:08:40.998769 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:40.998746 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 03 14:08:41.000998 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.000972 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dplmq" Jul 03 14:08:41.003374 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.003183 2576 manager.go:354] Recovery completed Jul 03 14:08:41.008941 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.008927 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 14:08:41.010177 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.010157 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dplmq" Jul 03 14:08:41.011014 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.010997 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasSufficientMemory" Jul 03 14:08:41.011116 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.011028 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 14:08:41.011116 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.011043 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasSufficientPID" Jul 03 14:08:41.011649 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.011631 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 03 14:08:41.011710 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.011650 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 03 14:08:41.011710 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.011669 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 03 14:08:41.011780 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.011641 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 14:08:41.012534 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.012520 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasSufficientMemory" Jul 03 14:08:41.012584 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.012546 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 14:08:41.012584 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.012556 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasSufficientPID" Jul 03 14:08:41.013215 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.013202 2576 policy_none.go:49] "None policy: Start" Jul 03 14:08:41.013254 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.013228 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 03 14:08:41.013254 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.013243 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 03 14:08:41.013342 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:41.013171 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-82.ec2.internal.18beccd5c638eb83 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-82.ec2.internal,UID:ip-10-0-140-82.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-82.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-82.ec2.internal,},FirstTimestamp:2026-07-03 14:08:41.011014531 +0000 UTC m=+0.402692251,LastTimestamp:2026-07-03 14:08:41.011014531 +0000 UTC m=+0.402692251,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-82.ec2.internal,}" Jul 03 14:08:41.014751 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.014739 2576 policy_none.go:47] "Start" Jul 03 14:08:41.051934 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.051913 2576 manager.go:337] "Starting Device Plugin manager" Jul 03 14:08:41.052081 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:41.051949 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 03 14:08:41.052081 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.051960 2576 server.go:87] "Starting device plugin registration server" Jul 03 14:08:41.052288 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.052256 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 03 14:08:41.052344 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.052293 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 03 14:08:41.052411 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.052396 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 03 14:08:41.052498 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.052481 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 03 14:08:41.052498 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.052495 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 03 14:08:41.053056 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:41.053041 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 03 14:08:41.053125 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:41.053085 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-82.ec2.internal\" not found" Jul 03 14:08:41.124670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.124625 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 03 14:08:41.126012 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.125988 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 03 14:08:41.126101 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.126027 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 03 14:08:41.126101 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.126055 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 03 14:08:41.126166 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:41.126100 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 03 14:08:41.128615 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.128591 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 14:08:41.152598 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.152561 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 14:08:41.153588 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.153558 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasSufficientMemory" Jul 03 14:08:41.153698 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.153597 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 14:08:41.153698 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.153607 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasSufficientPID" Jul 03 14:08:41.153698 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.153634 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-82.ec2.internal" Jul 03 14:08:41.165143 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.165086 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-82.ec2.internal" Jul 03 14:08:41.165143 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:41.165113 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-82.ec2.internal\": node \"ip-10-0-140-82.ec2.internal\" not found" Jul 03 14:08:41.183596 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:41.183573 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-82.ec2.internal\" not found" Jul 03 14:08:41.226553 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.226524 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-82.ec2.internal"] Jul 03 14:08:41.226637 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.226600 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 14:08:41.228223 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.228205 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasSufficientMemory" Jul 03 14:08:41.228337 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.228238 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 14:08:41.228337 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.228248 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasSufficientPID" Jul 03 14:08:41.229759 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.229744 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 14:08:41.229901 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.229886 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal" Jul 03 14:08:41.229963 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.229922 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 14:08:41.230514 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.230499 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasSufficientMemory" Jul 03 14:08:41.230514 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.230509 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasSufficientMemory" Jul 03 14:08:41.230612 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.230530 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 14:08:41.230612 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.230546 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasSufficientPID" Jul 03 14:08:41.230612 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.230532 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 14:08:41.230699 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.230622 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasSufficientPID" Jul 03 14:08:41.231988 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.231969 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-82.ec2.internal" Jul 03 14:08:41.232067 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.231996 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 14:08:41.232690 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.232675 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasSufficientMemory" Jul 03 14:08:41.232768 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.232700 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 14:08:41.232768 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:41.232709 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeHasSufficientPID" Jul 03 14:08:41.263103 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:41.263078 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-82.ec2.internal\" not found" node="ip-10-0-140-82.ec2.internal" Jul 03 14:08:41.267732 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:41.267714 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-82.ec2.internal\" not found" node="ip-10-0-140-82.ec2.internal" Jul 03 14:08:42.319273 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:42.319251 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-82.ec2.internal\" not found" Jul 03 14:08:42.330126 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:42.330097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aabef1e1070cbce5457cd456994a35fb-config\") pod \"kube-apiserver-proxy-ip-10-0-140-82.ec2.internal\" (UID: \"aabef1e1070cbce5457cd456994a35fb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-82.ec2.internal" Jul 03 14:08:42.330226 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:42.330128 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea23e504b34f7d7760470a819fbfdb46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal\" (UID: \"ea23e504b34f7d7760470a819fbfdb46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal" Jul 03 14:08:42.330226 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:42.330146 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea23e504b34f7d7760470a819fbfdb46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal\" (UID: \"ea23e504b34f7d7760470a819fbfdb46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal" Jul 03 14:08:42.419734 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:42.419704 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-82.ec2.internal\" not found" Jul 03 14:08:42.431111 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:42.431085 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aabef1e1070cbce5457cd456994a35fb-config\") pod \"kube-apiserver-proxy-ip-10-0-140-82.ec2.internal\" (UID: \"aabef1e1070cbce5457cd456994a35fb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-82.ec2.internal" Jul 03 14:08:42.431219 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:42.431117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea23e504b34f7d7760470a819fbfdb46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal\" (UID: \"ea23e504b34f7d7760470a819fbfdb46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal" Jul 03 14:08:42.431219 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:42.431138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea23e504b34f7d7760470a819fbfdb46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal\" (UID: \"ea23e504b34f7d7760470a819fbfdb46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal" Jul 03 14:08:42.431219 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:42.431180 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aabef1e1070cbce5457cd456994a35fb-config\") pod \"kube-apiserver-proxy-ip-10-0-140-82.ec2.internal\" (UID: \"aabef1e1070cbce5457cd456994a35fb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-82.ec2.internal" Jul 03 14:08:42.431219 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:42.431203 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea23e504b34f7d7760470a819fbfdb46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal\" (UID: \"ea23e504b34f7d7760470a819fbfdb46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal" Jul 03 14:08:42.431219 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:42.431196 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea23e504b34f7d7760470a819fbfdb46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal\" (UID: \"ea23e504b34f7d7760470a819fbfdb46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal" Jul 03 14:08:42.520619 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:42.520542 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-82.ec2.internal\" not found" Jul 03 14:08:42.602152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:42.602113 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal" Jul 03 14:08:42.606561 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:42.606080 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-82.ec2.internal" Jul 03 14:08:42.621575 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:42.621550 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-82.ec2.internal\" not found" Jul 03 14:08:42.722131 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:42.722096 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-82.ec2.internal\" not found" Jul 03 14:08:42.822632 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:42.822589 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-82.ec2.internal\" not found" Jul 03 14:08:42.923191 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:42.923149 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-82.ec2.internal\" not found" Jul 03 14:08:42.975631 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:42.975608 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 03 14:08:42.976077 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:42.975747 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 03 14:08:43.023261 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:43.023237 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-82.ec2.internal\" not found" Jul 03 14:08:43.050302 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:43.050265 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-02 14:03:42 +0000 UTC" deadline="2028-02-22 03:37:36.221551692 +0000 UTC" Jul 03 14:08:43.050302 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:43.050300 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14365h28m53.171255077s" Jul 03 14:08:43.053719 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:43.053703 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 14:08:43.123680 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:43.123589 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-82.ec2.internal\" not found" Jul 03 14:08:43.211748 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:43.211729 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 14:08:43.228318 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:43.228296 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal" Jul 03 14:08:43.244648 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:43.244625 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 03 14:08:43.245604 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:43.245590 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-82.ec2.internal" Jul 03 14:08:43.256549 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:43.256018 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 03 14:08:43.322435 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:43.322393 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea23e504b34f7d7760470a819fbfdb46.slice/crio-50bd1eb2b0df894c7c8ca316fa4ebcee94d01cb0d08690a1bb58664b3d271b40 WatchSource:0}: Error finding container 50bd1eb2b0df894c7c8ca316fa4ebcee94d01cb0d08690a1bb58664b3d271b40: Status 404 returned error can't find the container with id 50bd1eb2b0df894c7c8ca316fa4ebcee94d01cb0d08690a1bb58664b3d271b40 Jul 03 14:08:43.322900 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:43.322877 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaabef1e1070cbce5457cd456994a35fb.slice/crio-84192447c10aa0caec573e36ea810198430cbf40788ea3e2580f3838d9259d71 WatchSource:0}: Error finding container 84192447c10aa0caec573e36ea810198430cbf40788ea3e2580f3838d9259d71: Status 404 returned error can't find the container with id 84192447c10aa0caec573e36ea810198430cbf40788ea3e2580f3838d9259d71 Jul 03 14:08:43.327403 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:43.327387 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 03 14:08:43.428702 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:43.428610 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 14:08:44.010055 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.010026 2576 apiserver.go:52] "Watching apiserver" Jul 03 14:08:44.022257 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.022228 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 03 14:08:44.022765 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.022735 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-vtmd7","kube-system/global-pull-secret-syncer-xn7tc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5","openshift-cluster-node-tuning-operator/tuned-v2kxm","openshift-image-registry/node-ca-hjwhk","openshift-multus/network-metrics-daemon-7n9kl","openshift-network-diagnostics/network-check-target-6n5dr","kube-system/konnectivity-agent-lbt8h","kube-system/kube-apiserver-proxy-ip-10-0-140-82.ec2.internal","openshift-dns/node-resolver-4v7fc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal","openshift-multus/multus-7w6pw","openshift-multus/multus-additional-cni-plugins-fvf88","openshift-network-operator/iptables-alerter-ksn4z"] Jul 03 14:08:44.024810 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.024457 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:08:44.024810 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:44.024533 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6n5dr" podUID="e48e7e11-89d3-4f42-be11-8f2b3c685504" Jul 03 14:08:44.025901 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.025879 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.027514 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.027492 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.028003 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.027983 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 03 14:08:44.029080 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.029015 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.030275 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.030254 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 03 14:08:44.030563 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.030543 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 03 14:08:44.030833 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.030809 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 03 14:08:44.031268 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.031250 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 03 14:08:44.033122 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.031462 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 03 14:08:44.033122 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.031729 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dmrfl\"" Jul 03 14:08:44.033122 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.032173 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ngmrj\"" Jul 03 14:08:44.033122 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.032207 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 03 14:08:44.033122 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.032407 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 03 14:08:44.033122 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.032416 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 03 14:08:44.033122 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.032579 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 03 14:08:44.033122 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.032607 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 03 14:08:44.033122 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.032932 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7bkql\"" Jul 03 14:08:44.033718 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.033701 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hjwhk" Jul 03 14:08:44.035187 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.035163 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:08:44.035284 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:44.035228 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n9kl" podUID="514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb" Jul 03 14:08:44.036167 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.036144 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 03 14:08:44.036260 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.036171 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4jlm2\"" Jul 03 14:08:44.036305 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.036150 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 03 14:08:44.036457 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.036443 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 03 14:08:44.036857 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.036842 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ksn4z" Jul 03 14:08:44.038235 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.038107 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.038778 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.038758 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5515218d-73b7-4e7f-bb75-cddca2c8a06a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.038855 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.038791 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-etc-modprobe-d\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.038855 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.038812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-host\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.038855 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.038837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae42d692-7188-4fc2-b540-afebb1979245-etc-tuned\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.039013 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.038864 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-system-cni-dir\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.039013 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.038893 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-cnibin\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.039013 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.038917 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-cni-binary-copy\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.039013 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.038974 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-lib-modules\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.039013 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039003 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.039243 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039025 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5515218d-73b7-4e7f-bb75-cddca2c8a06a-sys-fs\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.039243 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039049 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-etc-sysctl-conf\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.039243 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-etc-systemd\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.039243 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039096 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-os-release\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.039243 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039118 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-var-lib-kubelet\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.039243 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039140 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae42d692-7188-4fc2-b540-afebb1979245-tmp\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.039243 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8shs\" (UniqueName: \"kubernetes.io/projected/ae42d692-7188-4fc2-b540-afebb1979245-kube-api-access-g8shs\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.039243 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnsvt\" (UniqueName: \"kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt\") pod \"network-check-target-6n5dr\" (UID: \"e48e7e11-89d3-4f42-be11-8f2b3c685504\") " pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:08:44.039243 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039220 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-run\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.039243 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039243 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-sys\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.039715 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039280 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.039715 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039304 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 03 14:08:44.039715 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039305 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.039715 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039338 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5515218d-73b7-4e7f-bb75-cddca2c8a06a-registration-dir\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.039715 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039362 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5515218d-73b7-4e7f-bb75-cddca2c8a06a-device-dir\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.039715 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5515218d-73b7-4e7f-bb75-cddca2c8a06a-etc-selinux\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.039715 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039412 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8g8r\" (UniqueName: \"kubernetes.io/projected/5515218d-73b7-4e7f-bb75-cddca2c8a06a-kube-api-access-n8g8r\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.039715 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039438 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 03 14:08:44.039715 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039443 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-etc-kubernetes\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.039715 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krp76\" (UniqueName: \"kubernetes.io/projected/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-kube-api-access-krp76\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.039715 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5515218d-73b7-4e7f-bb75-cddca2c8a06a-socket-dir\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.039715 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-etc-sysconfig\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.039715 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-etc-sysctl-d\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.039715 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039579 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xhkk2\"" Jul 03 14:08:44.040331 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.039752 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 03 14:08:44.040331 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.040292 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:44.040435 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:44.040344 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xn7tc" podUID="21624337-43ef-4f45-8118-63b9de2f9ed9" Jul 03 14:08:44.041259 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.041237 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 03 14:08:44.041487 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.041469 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 03 14:08:44.042245 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.042224 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zhrms\"" Jul 03 14:08:44.042437 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.042419 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 03 14:08:44.042811 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.042789 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 03 14:08:44.042927 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.042911 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 03 14:08:44.042993 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.042919 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 03 14:08:44.042993 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.042957 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 03 14:08:44.043096 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.043072 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4v7fc" Jul 03 14:08:44.043503 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.043218 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lbt8h" Jul 03 14:08:44.045638 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.045562 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.046286 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.046266 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 03 14:08:44.046639 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.046620 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mgdx7\"" Jul 03 14:08:44.046854 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.046838 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 03 14:08:44.046960 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.046941 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 03 14:08:44.047523 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.047498 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-fsfl7\"" Jul 03 14:08:44.047764 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.047751 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 03 14:08:44.048612 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.048595 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 03 14:08:44.048715 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.048609 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lds9x\"" Jul 03 14:08:44.073041 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.073016 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zgxx4" Jul 03 14:08:44.082362 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.082339 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zgxx4" Jul 03 14:08:44.128651 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.128621 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 03 14:08:44.140724 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.140697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-host\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.140724 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.140726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae42d692-7188-4fc2-b540-afebb1979245-etc-tuned\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.140887 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.140748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-run-ovn-kubernetes\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.140887 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.140771 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-host-run-netns\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.140887 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.140793 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c77a739f-276a-4602-824b-874f70e83982-multus-daemon-config\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.140887 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.140816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-cnibin\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.140887 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.140838 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-cni-binary-copy\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.140887 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.140863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5f6cf69-3f8b-4afe-a474-4dffa0448079-env-overrides\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.140887 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.140877 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-cnibin\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.141168 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.140884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/988adf65-e4cb-4b10-ae2c-f689170cf998-host-slash\") pod \"iptables-alerter-ksn4z\" (UID: \"988adf65-e4cb-4b10-ae2c-f689170cf998\") " pod="openshift-network-operator/iptables-alerter-ksn4z" Jul 03 14:08:44.141168 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.140935 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr98z\" (UniqueName: \"kubernetes.io/projected/988adf65-e4cb-4b10-ae2c-f689170cf998-kube-api-access-xr98z\") pod \"iptables-alerter-ksn4z\" (UID: \"988adf65-e4cb-4b10-ae2c-f689170cf998\") " pod="openshift-network-operator/iptables-alerter-ksn4z" Jul 03 14:08:44.141168 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.140955 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99aa3d88-a61c-4e92-adc3-a69b48070430-agent-certs\") pod \"konnectivity-agent-lbt8h\" (UID: \"99aa3d88-a61c-4e92-adc3-a69b48070430\") " pod="kube-system/konnectivity-agent-lbt8h" Jul 03 14:08:44.141168 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.140971 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-multus-cni-dir\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.141168 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.140987 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/988adf65-e4cb-4b10-ae2c-f689170cf998-iptables-alerter-script\") pod \"iptables-alerter-ksn4z\" (UID: \"988adf65-e4cb-4b10-ae2c-f689170cf998\") " pod="openshift-network-operator/iptables-alerter-ksn4z" Jul 03 14:08:44.141168 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-multus-conf-dir\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.141168 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-etc-kubernetes\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.141168 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141028 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcpqf\" (UniqueName: \"kubernetes.io/projected/c77a739f-276a-4602-824b-874f70e83982-kube-api-access-wcpqf\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.141168 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.141168 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141063 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5515218d-73b7-4e7f-bb75-cddca2c8a06a-sys-fs\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.141168 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.140839 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-host\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.141168 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141077 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-run-systemd\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.141168 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141092 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 03 14:08:44.141168 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141140 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-var-lib-openvswitch\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.141928 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141180 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs\") pod \"network-metrics-daemon-7n9kl\" (UID: \"514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb\") " pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:08:44.141928 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141210 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-log-socket\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.141928 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141218 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5515218d-73b7-4e7f-bb75-cddca2c8a06a-sys-fs\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.141928 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141237 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm922\" (UniqueName: \"kubernetes.io/projected/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-kube-api-access-rm922\") pod \"network-metrics-daemon-7n9kl\" (UID: \"514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb\") " pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:08:44.141928 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-sys\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.141928 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141294 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-kubelet\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.141928 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.141928 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141351 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.141928 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141377 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.141928 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5515218d-73b7-4e7f-bb75-cddca2c8a06a-device-dir\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.141928 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141426 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-etc-kubernetes\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.141928 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141444 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-cni-binary-copy\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.141928 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141471 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-sys\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.141928 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/928b4e88-1d06-41e9-99c9-efecc91c0ee9-hosts-file\") pod \"node-resolver-4v7fc\" (UID: \"928b4e88-1d06-41e9-99c9-efecc91c0ee9\") " pod="openshift-dns/node-resolver-4v7fc" Jul 03 14:08:44.141928 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141514 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-system-cni-dir\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.141928 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141534 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5515218d-73b7-4e7f-bb75-cddca2c8a06a-device-dir\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.141928 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-etc-sysctl-d\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.142858 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141571 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-run-ovn\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.142858 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141584 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-etc-kubernetes\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.142858 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141590 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.142858 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141596 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-cni-netd\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.142858 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99aa3d88-a61c-4e92-adc3-a69b48070430-konnectivity-ca\") pod \"konnectivity-agent-lbt8h\" (UID: \"99aa3d88-a61c-4e92-adc3-a69b48070430\") " pod="kube-system/konnectivity-agent-lbt8h" Jul 03 14:08:44.142858 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141641 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-cnibin\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.142858 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141683 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-os-release\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.142858 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5515218d-73b7-4e7f-bb75-cddca2c8a06a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.142858 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141718 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-etc-sysctl-d\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.142858 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141733 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-etc-modprobe-d\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.142858 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141751 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.142858 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141761 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-etc-openvswitch\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.142858 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141779 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-cni-bin\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.142858 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141793 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5f6cf69-3f8b-4afe-a474-4dffa0448079-ovnkube-script-lib\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.142858 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141796 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5515218d-73b7-4e7f-bb75-cddca2c8a06a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.142858 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-system-cni-dir\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.142858 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141836 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b6e59bf8-184d-4593-aa5f-de1c5854ac77-serviceca\") pod \"node-ca-hjwhk\" (UID: \"b6e59bf8-184d-4593-aa5f-de1c5854ac77\") " pod="openshift-image-registry/node-ca-hjwhk" Jul 03 14:08:44.143620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141850 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-etc-modprobe-d\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.143620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-systemd-units\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.143620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141891 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-node-log\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.143620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141896 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-system-cni-dir\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.143620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-lib-modules\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.143620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141962 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.143620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.141990 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-lib-modules\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.143620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142016 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-host-var-lib-cni-multus\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.143620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142043 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-host-var-lib-kubelet\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.143620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-etc-sysctl-conf\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.143620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142095 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-etc-systemd\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.143620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-slash\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.143620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142141 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-run-openvswitch\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.143620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142150 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-etc-systemd\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.143620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928b4e88-1d06-41e9-99c9-efecc91c0ee9-tmp-dir\") pod \"node-resolver-4v7fc\" (UID: \"928b4e88-1d06-41e9-99c9-efecc91c0ee9\") " pod="openshift-dns/node-resolver-4v7fc" Jul 03 14:08:44.143620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c77a739f-276a-4602-824b-874f70e83982-cni-binary-copy\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.143620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142195 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-etc-sysctl-conf\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.144318 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142213 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-host-run-k8s-cni-cncf-io\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.144318 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142237 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-hostroot\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.144318 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-os-release\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.144318 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-var-lib-kubelet\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.144318 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142311 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-os-release\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.144318 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142324 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae42d692-7188-4fc2-b540-afebb1979245-tmp\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.144318 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8shs\" (UniqueName: \"kubernetes.io/projected/ae42d692-7188-4fc2-b540-afebb1979245-kube-api-access-g8shs\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.144318 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142367 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-var-lib-kubelet\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.144318 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7srj\" (UniqueName: \"kubernetes.io/projected/c5f6cf69-3f8b-4afe-a474-4dffa0448079-kube-api-access-t7srj\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.144318 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21624337-43ef-4f45-8118-63b9de2f9ed9-dbus\") pod \"global-pull-secret-syncer-xn7tc\" (UID: \"21624337-43ef-4f45-8118-63b9de2f9ed9\") " pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:44.144318 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-host-run-multus-certs\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.144318 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnsvt\" (UniqueName: \"kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt\") pod \"network-check-target-6n5dr\" (UID: \"e48e7e11-89d3-4f42-be11-8f2b3c685504\") " pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:08:44.144318 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-run\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.144318 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142596 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5f6cf69-3f8b-4afe-a474-4dffa0448079-ovn-node-metrics-cert\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.144318 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret\") pod \"global-pull-secret-syncer-xn7tc\" (UID: \"21624337-43ef-4f45-8118-63b9de2f9ed9\") " pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:44.144318 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142647 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w52zx\" (UniqueName: \"kubernetes.io/projected/b6e59bf8-184d-4593-aa5f-de1c5854ac77-kube-api-access-w52zx\") pod \"node-ca-hjwhk\" (UID: \"b6e59bf8-184d-4593-aa5f-de1c5854ac77\") " pod="openshift-image-registry/node-ca-hjwhk" Jul 03 14:08:44.144318 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142672 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-run\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.144989 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142687 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5f6cf69-3f8b-4afe-a474-4dffa0448079-ovnkube-config\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.144989 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5515218d-73b7-4e7f-bb75-cddca2c8a06a-registration-dir\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.144989 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5515218d-73b7-4e7f-bb75-cddca2c8a06a-etc-selinux\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.144989 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8g8r\" (UniqueName: \"kubernetes.io/projected/5515218d-73b7-4e7f-bb75-cddca2c8a06a-kube-api-access-n8g8r\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.144989 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142820 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5515218d-73b7-4e7f-bb75-cddca2c8a06a-registration-dir\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.144989 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6e59bf8-184d-4593-aa5f-de1c5854ac77-host\") pod \"node-ca-hjwhk\" (UID: \"b6e59bf8-184d-4593-aa5f-de1c5854ac77\") " pod="openshift-image-registry/node-ca-hjwhk" Jul 03 14:08:44.144989 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142851 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drncf\" (UniqueName: \"kubernetes.io/projected/928b4e88-1d06-41e9-99c9-efecc91c0ee9-kube-api-access-drncf\") pod \"node-resolver-4v7fc\" (UID: \"928b4e88-1d06-41e9-99c9-efecc91c0ee9\") " pod="openshift-dns/node-resolver-4v7fc" Jul 03 14:08:44.144989 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142873 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21624337-43ef-4f45-8118-63b9de2f9ed9-kubelet-config\") pod \"global-pull-secret-syncer-xn7tc\" (UID: \"21624337-43ef-4f45-8118-63b9de2f9ed9\") " pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:44.144989 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krp76\" (UniqueName: \"kubernetes.io/projected/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-kube-api-access-krp76\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.144989 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142905 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5515218d-73b7-4e7f-bb75-cddca2c8a06a-etc-selinux\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.144989 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5515218d-73b7-4e7f-bb75-cddca2c8a06a-socket-dir\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.144989 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-etc-sysconfig\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.144989 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.142986 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-run-netns\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.144989 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.143013 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-multus-socket-dir-parent\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.144989 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.143037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-host-var-lib-cni-bin\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.144989 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.143060 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae42d692-7188-4fc2-b540-afebb1979245-etc-sysconfig\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.145844 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.143067 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5515218d-73b7-4e7f-bb75-cddca2c8a06a-socket-dir\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.145844 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.144123 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae42d692-7188-4fc2-b540-afebb1979245-etc-tuned\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.145844 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.144521 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae42d692-7188-4fc2-b540-afebb1979245-tmp\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.149416 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:44.149393 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 14:08:44.149511 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:44.149416 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 14:08:44.149511 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:44.149433 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cnsvt for pod openshift-network-diagnostics/network-check-target-6n5dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 14:08:44.149622 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:44.149515 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt podName:e48e7e11-89d3-4f42-be11-8f2b3c685504 nodeName:}" failed. No retries permitted until 2026-07-03 14:08:44.649483895 +0000 UTC m=+3.005768512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cnsvt" (UniqueName: "kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt") pod "network-check-target-6n5dr" (UID: "e48e7e11-89d3-4f42-be11-8f2b3c685504") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 14:08:44.151827 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.151650 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8shs\" (UniqueName: \"kubernetes.io/projected/ae42d692-7188-4fc2-b540-afebb1979245-kube-api-access-g8shs\") pod \"tuned-v2kxm\" (UID: \"ae42d692-7188-4fc2-b540-afebb1979245\") " pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.152039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.152017 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8g8r\" (UniqueName: \"kubernetes.io/projected/5515218d-73b7-4e7f-bb75-cddca2c8a06a-kube-api-access-n8g8r\") pod \"aws-ebs-csi-driver-node-lflw5\" (UID: \"5515218d-73b7-4e7f-bb75-cddca2c8a06a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.152301 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.152279 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krp76\" (UniqueName: \"kubernetes.io/projected/f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d-kube-api-access-krp76\") pod \"multus-additional-cni-plugins-fvf88\" (UID: \"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d\") " pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.169159 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.169028 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-82.ec2.internal" event={"ID":"aabef1e1070cbce5457cd456994a35fb","Type":"ContainerStarted","Data":"84192447c10aa0caec573e36ea810198430cbf40788ea3e2580f3838d9259d71"} Jul 03 14:08:44.170559 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.170533 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal" event={"ID":"ea23e504b34f7d7760470a819fbfdb46","Type":"ContainerStarted","Data":"50bd1eb2b0df894c7c8ca316fa4ebcee94d01cb0d08690a1bb58664b3d271b40"} Jul 03 14:08:44.243604 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.243529 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-kubelet\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.243604 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.243570 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.243604 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.243600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/928b4e88-1d06-41e9-99c9-efecc91c0ee9-hosts-file\") pod \"node-resolver-4v7fc\" (UID: \"928b4e88-1d06-41e9-99c9-efecc91c0ee9\") " pod="openshift-dns/node-resolver-4v7fc" Jul 03 14:08:44.243876 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.243625 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-system-cni-dir\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.243876 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.243649 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-run-ovn\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.243876 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.243716 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-cni-netd\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.243876 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.243741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99aa3d88-a61c-4e92-adc3-a69b48070430-konnectivity-ca\") pod \"konnectivity-agent-lbt8h\" (UID: \"99aa3d88-a61c-4e92-adc3-a69b48070430\") " pod="kube-system/konnectivity-agent-lbt8h" Jul 03 14:08:44.243876 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.243763 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-cnibin\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.243876 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.243787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-os-release\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.243876 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.243839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-etc-openvswitch\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.243876 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.243854 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/928b4e88-1d06-41e9-99c9-efecc91c0ee9-hosts-file\") pod \"node-resolver-4v7fc\" (UID: \"928b4e88-1d06-41e9-99c9-efecc91c0ee9\") " pod="openshift-dns/node-resolver-4v7fc" Jul 03 14:08:44.243876 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.243863 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-cni-bin\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.244274 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.243895 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5f6cf69-3f8b-4afe-a474-4dffa0448079-ovnkube-script-lib\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.244274 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.243918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b6e59bf8-184d-4593-aa5f-de1c5854ac77-serviceca\") pod \"node-ca-hjwhk\" (UID: \"b6e59bf8-184d-4593-aa5f-de1c5854ac77\") " pod="openshift-image-registry/node-ca-hjwhk" Jul 03 14:08:44.244274 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.243954 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-systemd-units\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.244274 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.243978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-node-log\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.244274 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-host-var-lib-cni-multus\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.244274 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244011 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-system-cni-dir\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.244274 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244035 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-host-var-lib-kubelet\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.244274 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244064 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-run-ovn\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.244274 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244069 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-slash\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.244274 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-run-openvswitch\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.244274 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244137 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-slash\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.244274 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928b4e88-1d06-41e9-99c9-efecc91c0ee9-tmp-dir\") pod \"node-resolver-4v7fc\" (UID: \"928b4e88-1d06-41e9-99c9-efecc91c0ee9\") " pod="openshift-dns/node-resolver-4v7fc" Jul 03 14:08:44.244274 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244180 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-etc-openvswitch\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.244274 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c77a739f-276a-4602-824b-874f70e83982-cni-binary-copy\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.244274 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-host-run-k8s-cni-cncf-io\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.244274 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244239 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-hostroot\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.244274 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244266 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7srj\" (UniqueName: \"kubernetes.io/projected/c5f6cf69-3f8b-4afe-a474-4dffa0448079-kube-api-access-t7srj\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.244817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244299 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21624337-43ef-4f45-8118-63b9de2f9ed9-dbus\") pod \"global-pull-secret-syncer-xn7tc\" (UID: \"21624337-43ef-4f45-8118-63b9de2f9ed9\") " pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:44.244817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244323 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-host-run-multus-certs\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.244817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5f6cf69-3f8b-4afe-a474-4dffa0448079-ovn-node-metrics-cert\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.244817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244384 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret\") pod \"global-pull-secret-syncer-xn7tc\" (UID: \"21624337-43ef-4f45-8118-63b9de2f9ed9\") " pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:44.244817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244461 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w52zx\" (UniqueName: \"kubernetes.io/projected/b6e59bf8-184d-4593-aa5f-de1c5854ac77-kube-api-access-w52zx\") pod \"node-ca-hjwhk\" (UID: \"b6e59bf8-184d-4593-aa5f-de1c5854ac77\") " pod="openshift-image-registry/node-ca-hjwhk" Jul 03 14:08:44.244817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244476 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5f6cf69-3f8b-4afe-a474-4dffa0448079-ovnkube-config\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.244817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244500 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6e59bf8-184d-4593-aa5f-de1c5854ac77-host\") pod \"node-ca-hjwhk\" (UID: \"b6e59bf8-184d-4593-aa5f-de1c5854ac77\") " pod="openshift-image-registry/node-ca-hjwhk" Jul 03 14:08:44.244817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244514 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drncf\" (UniqueName: \"kubernetes.io/projected/928b4e88-1d06-41e9-99c9-efecc91c0ee9-kube-api-access-drncf\") pod \"node-resolver-4v7fc\" (UID: \"928b4e88-1d06-41e9-99c9-efecc91c0ee9\") " pod="openshift-dns/node-resolver-4v7fc" Jul 03 14:08:44.244817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244534 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21624337-43ef-4f45-8118-63b9de2f9ed9-kubelet-config\") pod \"global-pull-secret-syncer-xn7tc\" (UID: \"21624337-43ef-4f45-8118-63b9de2f9ed9\") " pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:44.244817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244552 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-run-netns\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.244817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-multus-socket-dir-parent\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.244817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-host-var-lib-cni-bin\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.244817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244672 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c77a739f-276a-4602-824b-874f70e83982-cni-binary-copy\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.244817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-run-ovn-kubernetes\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.244817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-host-run-netns\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.244817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244742 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c77a739f-276a-4602-824b-874f70e83982-multus-daemon-config\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.244817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244759 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-cnibin\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.244817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244776 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5f6cf69-3f8b-4afe-a474-4dffa0448079-env-overrides\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.245601 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244772 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-cni-netd\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.245601 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244834 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21624337-43ef-4f45-8118-63b9de2f9ed9-kubelet-config\") pod \"global-pull-secret-syncer-xn7tc\" (UID: \"21624337-43ef-4f45-8118-63b9de2f9ed9\") " pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:44.245601 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244884 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-host-var-lib-cni-bin\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.245601 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244886 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-host-run-netns\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.245601 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.245030 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21624337-43ef-4f45-8118-63b9de2f9ed9-dbus\") pod \"global-pull-secret-syncer-xn7tc\" (UID: \"21624337-43ef-4f45-8118-63b9de2f9ed9\") " pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:44.245601 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.245062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/988adf65-e4cb-4b10-ae2c-f689170cf998-host-slash\") pod \"iptables-alerter-ksn4z\" (UID: \"988adf65-e4cb-4b10-ae2c-f689170cf998\") " pod="openshift-network-operator/iptables-alerter-ksn4z" Jul 03 14:08:44.245601 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.245091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr98z\" (UniqueName: \"kubernetes.io/projected/988adf65-e4cb-4b10-ae2c-f689170cf998-kube-api-access-xr98z\") pod \"iptables-alerter-ksn4z\" (UID: \"988adf65-e4cb-4b10-ae2c-f689170cf998\") " pod="openshift-network-operator/iptables-alerter-ksn4z" Jul 03 14:08:44.245601 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.245117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99aa3d88-a61c-4e92-adc3-a69b48070430-agent-certs\") pod \"konnectivity-agent-lbt8h\" (UID: \"99aa3d88-a61c-4e92-adc3-a69b48070430\") " pod="kube-system/konnectivity-agent-lbt8h" Jul 03 14:08:44.245601 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.245143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-multus-cni-dir\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.245601 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.245171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/988adf65-e4cb-4b10-ae2c-f689170cf998-iptables-alerter-script\") pod \"iptables-alerter-ksn4z\" (UID: \"988adf65-e4cb-4b10-ae2c-f689170cf998\") " pod="openshift-network-operator/iptables-alerter-ksn4z" Jul 03 14:08:44.245601 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.245195 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-multus-conf-dir\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.245601 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.245211 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-host-run-multus-certs\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.245601 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.245223 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-etc-kubernetes\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.245601 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.245240 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928b4e88-1d06-41e9-99c9-efecc91c0ee9-tmp-dir\") pod \"node-resolver-4v7fc\" (UID: \"928b4e88-1d06-41e9-99c9-efecc91c0ee9\") " pod="openshift-dns/node-resolver-4v7fc" Jul 03 14:08:44.245601 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.245249 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcpqf\" (UniqueName: \"kubernetes.io/projected/c77a739f-276a-4602-824b-874f70e83982-kube-api-access-wcpqf\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.245601 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:44.245286 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 14:08:44.245601 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.245285 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b6e59bf8-184d-4593-aa5f-de1c5854ac77-serviceca\") pod \"node-ca-hjwhk\" (UID: \"b6e59bf8-184d-4593-aa5f-de1c5854ac77\") " pod="openshift-image-registry/node-ca-hjwhk" Jul 03 14:08:44.245601 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.245288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-run-systemd\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.246488 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.243918 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-kubelet\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.246488 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.245354 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99aa3d88-a61c-4e92-adc3-a69b48070430-konnectivity-ca\") pod \"konnectivity-agent-lbt8h\" (UID: \"99aa3d88-a61c-4e92-adc3-a69b48070430\") " pod="kube-system/konnectivity-agent-lbt8h" Jul 03 14:08:44.246488 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.243953 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.246488 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.245807 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-node-log\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.246488 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.245915 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-os-release\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.246488 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.245872 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c77a739f-276a-4602-824b-874f70e83982-multus-daemon-config\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.246488 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.246379 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-cni-bin\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.246488 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244741 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-run-openvswitch\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.246805 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:44.246603 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret podName:21624337-43ef-4f45-8118-63b9de2f9ed9 nodeName:}" failed. No retries permitted until 2026-07-03 14:08:44.746568029 +0000 UTC m=+3.102852649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret") pod "global-pull-secret-syncer-xn7tc" (UID: "21624337-43ef-4f45-8118-63b9de2f9ed9") : object "kube-system"/"original-pull-secret" not registered Jul 03 14:08:44.246805 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.246649 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-var-lib-openvswitch\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.246947 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.246929 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-multus-socket-dir-parent\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.246947 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.244830 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-run-netns\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.247148 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.245855 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-systemd-units\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.247633 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.247361 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-host-run-ovn-kubernetes\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.247633 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.245323 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-run-systemd\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.247633 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.247436 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-multus-cni-dir\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.247633 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.247498 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-multus-conf-dir\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.247633 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.247568 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-etc-kubernetes\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.247954 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.247783 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5f6cf69-3f8b-4afe-a474-4dffa0448079-ovnkube-script-lib\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.247954 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.247939 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/988adf65-e4cb-4b10-ae2c-f689170cf998-iptables-alerter-script\") pod \"iptables-alerter-ksn4z\" (UID: \"988adf65-e4cb-4b10-ae2c-f689170cf998\") " pod="openshift-network-operator/iptables-alerter-ksn4z" Jul 03 14:08:44.248046 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.248029 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5f6cf69-3f8b-4afe-a474-4dffa0448079-env-overrides\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.248111 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.248088 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-host-var-lib-kubelet\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.248169 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.248148 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-host-var-lib-cni-multus\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.248451 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.248313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs\") pod \"network-metrics-daemon-7n9kl\" (UID: \"514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb\") " pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:08:44.248451 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.248352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-log-socket\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.248451 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.248398 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-var-lib-openvswitch\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.248451 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.248415 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5f6cf69-3f8b-4afe-a474-4dffa0448079-log-socket\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.248451 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.248448 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-host-run-k8s-cni-cncf-io\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.248730 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:44.248456 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 14:08:44.248730 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.248489 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6e59bf8-184d-4593-aa5f-de1c5854ac77-host\") pod \"node-ca-hjwhk\" (UID: \"b6e59bf8-184d-4593-aa5f-de1c5854ac77\") " pod="openshift-image-registry/node-ca-hjwhk" Jul 03 14:08:44.248730 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:44.248520 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs podName:514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb nodeName:}" failed. No retries permitted until 2026-07-03 14:08:44.74849901 +0000 UTC m=+3.104783623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs") pod "network-metrics-daemon-7n9kl" (UID: "514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 14:08:44.248730 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.248525 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c77a739f-276a-4602-824b-874f70e83982-hostroot\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.248730 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.248544 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm922\" (UniqueName: \"kubernetes.io/projected/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-kube-api-access-rm922\") pod \"network-metrics-daemon-7n9kl\" (UID: \"514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb\") " pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:08:44.248730 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.248624 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/988adf65-e4cb-4b10-ae2c-f689170cf998-host-slash\") pod \"iptables-alerter-ksn4z\" (UID: \"988adf65-e4cb-4b10-ae2c-f689170cf998\") " pod="openshift-network-operator/iptables-alerter-ksn4z" Jul 03 14:08:44.249030 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.249016 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99aa3d88-a61c-4e92-adc3-a69b48070430-agent-certs\") pod \"konnectivity-agent-lbt8h\" (UID: \"99aa3d88-a61c-4e92-adc3-a69b48070430\") " pod="kube-system/konnectivity-agent-lbt8h" Jul 03 14:08:44.249081 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.249033 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5f6cf69-3f8b-4afe-a474-4dffa0448079-ovnkube-config\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.252049 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.252021 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5f6cf69-3f8b-4afe-a474-4dffa0448079-ovn-node-metrics-cert\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.255195 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.255172 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7srj\" (UniqueName: \"kubernetes.io/projected/c5f6cf69-3f8b-4afe-a474-4dffa0448079-kube-api-access-t7srj\") pod \"ovnkube-node-vtmd7\" (UID: \"c5f6cf69-3f8b-4afe-a474-4dffa0448079\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.261796 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.261773 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drncf\" (UniqueName: \"kubernetes.io/projected/928b4e88-1d06-41e9-99c9-efecc91c0ee9-kube-api-access-drncf\") pod \"node-resolver-4v7fc\" (UID: \"928b4e88-1d06-41e9-99c9-efecc91c0ee9\") " pod="openshift-dns/node-resolver-4v7fc" Jul 03 14:08:44.262315 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.262282 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr98z\" (UniqueName: \"kubernetes.io/projected/988adf65-e4cb-4b10-ae2c-f689170cf998-kube-api-access-xr98z\") pod \"iptables-alerter-ksn4z\" (UID: \"988adf65-e4cb-4b10-ae2c-f689170cf998\") " pod="openshift-network-operator/iptables-alerter-ksn4z" Jul 03 14:08:44.262374 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.262285 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w52zx\" (UniqueName: \"kubernetes.io/projected/b6e59bf8-184d-4593-aa5f-de1c5854ac77-kube-api-access-w52zx\") pod \"node-ca-hjwhk\" (UID: \"b6e59bf8-184d-4593-aa5f-de1c5854ac77\") " pod="openshift-image-registry/node-ca-hjwhk" Jul 03 14:08:44.262484 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.262446 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm922\" (UniqueName: \"kubernetes.io/projected/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-kube-api-access-rm922\") pod \"network-metrics-daemon-7n9kl\" (UID: \"514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb\") " pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:08:44.263921 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.263900 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcpqf\" (UniqueName: \"kubernetes.io/projected/c77a739f-276a-4602-824b-874f70e83982-kube-api-access-wcpqf\") pod \"multus-7w6pw\" (UID: \"c77a739f-276a-4602-824b-874f70e83982\") " pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.289339 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.289313 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 14:08:44.342412 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.342377 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fvf88" Jul 03 14:08:44.349650 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.349625 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" Jul 03 14:08:44.361462 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.361441 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" Jul 03 14:08:44.367347 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.367327 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hjwhk" Jul 03 14:08:44.379157 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.379134 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ksn4z" Jul 03 14:08:44.386084 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.386063 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:08:44.393740 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.393720 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4v7fc" Jul 03 14:08:44.401444 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.401427 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lbt8h" Jul 03 14:08:44.407407 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.407387 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7w6pw" Jul 03 14:08:44.651631 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.651596 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnsvt\" (UniqueName: \"kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt\") pod \"network-check-target-6n5dr\" (UID: \"e48e7e11-89d3-4f42-be11-8f2b3c685504\") " pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:08:44.651807 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:44.651763 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 14:08:44.651807 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:44.651787 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 14:08:44.651807 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:44.651799 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cnsvt for pod openshift-network-diagnostics/network-check-target-6n5dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 14:08:44.651921 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:44.651862 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt podName:e48e7e11-89d3-4f42-be11-8f2b3c685504 nodeName:}" failed. No retries permitted until 2026-07-03 14:08:45.651847091 +0000 UTC m=+4.008131690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cnsvt" (UniqueName: "kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt") pod "network-check-target-6n5dr" (UID: "e48e7e11-89d3-4f42-be11-8f2b3c685504") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 14:08:44.752710 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.752675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret\") pod \"global-pull-secret-syncer-xn7tc\" (UID: \"21624337-43ef-4f45-8118-63b9de2f9ed9\") " pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:44.752874 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:44.752758 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs\") pod \"network-metrics-daemon-7n9kl\" (UID: \"514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb\") " pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:08:44.752874 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:44.752765 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 14:08:44.752874 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:44.752826 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret podName:21624337-43ef-4f45-8118-63b9de2f9ed9 nodeName:}" failed. No retries permitted until 2026-07-03 14:08:45.752809811 +0000 UTC m=+4.109094409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret") pod "global-pull-secret-syncer-xn7tc" (UID: "21624337-43ef-4f45-8118-63b9de2f9ed9") : object "kube-system"/"original-pull-secret" not registered Jul 03 14:08:44.753028 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:44.752875 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 14:08:44.753028 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:44.752923 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs podName:514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb nodeName:}" failed. No retries permitted until 2026-07-03 14:08:45.752906884 +0000 UTC m=+4.109191487 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs") pod "network-metrics-daemon-7n9kl" (UID: "514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 14:08:45.085178 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:45.085133 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-02 14:03:44 +0000 UTC" deadline="2028-02-05 16:37:20.759020067 +0000 UTC" Jul 03 14:08:45.085178 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:45.085173 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13970h28m35.673850987s" Jul 03 14:08:45.161911 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:45.161885 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:08:45.162076 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:45.161975 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6n5dr" podUID="e48e7e11-89d3-4f42-be11-8f2b3c685504" Jul 03 14:08:45.262232 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:45.262206 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928b4e88_1d06_41e9_99c9_efecc91c0ee9.slice/crio-268ca4a2d4a2eaea3d6933818b195c9898e6c5f26f9519184f38fef03ff42798 WatchSource:0}: Error finding container 268ca4a2d4a2eaea3d6933818b195c9898e6c5f26f9519184f38fef03ff42798: Status 404 returned error can't find the container with id 268ca4a2d4a2eaea3d6933818b195c9898e6c5f26f9519184f38fef03ff42798 Jul 03 14:08:45.263706 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:45.263688 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e59bf8_184d_4593_aa5f_de1c5854ac77.slice/crio-6f2b3d1b33e3180d3703e8dba7d41891aff0ec865791c69852352c736e30776b WatchSource:0}: Error finding container 6f2b3d1b33e3180d3703e8dba7d41891aff0ec865791c69852352c736e30776b: Status 404 returned error can't find the container with id 6f2b3d1b33e3180d3703e8dba7d41891aff0ec865791c69852352c736e30776b Jul 03 14:08:45.264401 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:45.264378 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99aa3d88_a61c_4e92_adc3_a69b48070430.slice/crio-1ffd0cc8186ec1891b254306351c9a0e7d1790951ae8f5f0176432bfff32a05c WatchSource:0}: Error finding container 1ffd0cc8186ec1891b254306351c9a0e7d1790951ae8f5f0176432bfff32a05c: Status 404 returned error can't find the container with id 1ffd0cc8186ec1891b254306351c9a0e7d1790951ae8f5f0176432bfff32a05c Jul 03 14:08:45.268043 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:45.268023 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d9de01_3ada_4cb0_ae41_49f2c3f70a5d.slice/crio-1b62d242a9025a3d26e2df7580db3ce56894a80227dae09202584d409b735d6a WatchSource:0}: Error finding container 1b62d242a9025a3d26e2df7580db3ce56894a80227dae09202584d409b735d6a: Status 404 returned error can't find the container with id 1b62d242a9025a3d26e2df7580db3ce56894a80227dae09202584d409b735d6a Jul 03 14:08:45.269174 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:45.269139 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5515218d_73b7_4e7f_bb75_cddca2c8a06a.slice/crio-a901ac346e6196156b4f19f8e8780fd8ec48d835eed7f344cb3c6181a66d454f WatchSource:0}: Error finding container a901ac346e6196156b4f19f8e8780fd8ec48d835eed7f344cb3c6181a66d454f: Status 404 returned error can't find the container with id a901ac346e6196156b4f19f8e8780fd8ec48d835eed7f344cb3c6181a66d454f Jul 03 14:08:45.271527 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:45.271403 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod988adf65_e4cb_4b10_ae2c_f689170cf998.slice/crio-09417d78ecd8be53d85f65ec8834dd444b779b083c5d3824b5b3ef4f78c01a1c WatchSource:0}: Error finding container 09417d78ecd8be53d85f65ec8834dd444b779b083c5d3824b5b3ef4f78c01a1c: Status 404 returned error can't find the container with id 09417d78ecd8be53d85f65ec8834dd444b779b083c5d3824b5b3ef4f78c01a1c Jul 03 14:08:45.272295 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:08:45.272267 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f6cf69_3f8b_4afe_a474_4dffa0448079.slice/crio-79c3bcd6b72ed20cec18ada7d2958494896e08dab4fb5e04dcb133463e411646 WatchSource:0}: Error finding container 79c3bcd6b72ed20cec18ada7d2958494896e08dab4fb5e04dcb133463e411646: Status 404 returned error can't find the container with id 79c3bcd6b72ed20cec18ada7d2958494896e08dab4fb5e04dcb133463e411646 Jul 03 14:08:45.659019 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:45.658738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnsvt\" (UniqueName: \"kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt\") pod \"network-check-target-6n5dr\" (UID: \"e48e7e11-89d3-4f42-be11-8f2b3c685504\") " pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:08:45.659180 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:45.658935 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 14:08:45.659180 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:45.659079 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 14:08:45.659180 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:45.659094 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cnsvt for pod openshift-network-diagnostics/network-check-target-6n5dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 14:08:45.659324 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:45.659223 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt podName:e48e7e11-89d3-4f42-be11-8f2b3c685504 nodeName:}" failed. No retries permitted until 2026-07-03 14:08:47.659203129 +0000 UTC m=+6.015487728 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cnsvt" (UniqueName: "kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt") pod "network-check-target-6n5dr" (UID: "e48e7e11-89d3-4f42-be11-8f2b3c685504") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 14:08:45.760189 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:45.760152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret\") pod \"global-pull-secret-syncer-xn7tc\" (UID: \"21624337-43ef-4f45-8118-63b9de2f9ed9\") " pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:45.760375 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:45.760230 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs\") pod \"network-metrics-daemon-7n9kl\" (UID: \"514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb\") " pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:08:45.760375 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:45.760364 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 14:08:45.760486 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:45.760428 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs podName:514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb nodeName:}" failed. No retries permitted until 2026-07-03 14:08:47.760409883 +0000 UTC m=+6.116694486 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs") pod "network-metrics-daemon-7n9kl" (UID: "514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 14:08:45.760545 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:45.760500 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 14:08:45.760599 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:45.760546 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret podName:21624337-43ef-4f45-8118-63b9de2f9ed9 nodeName:}" failed. No retries permitted until 2026-07-03 14:08:47.760532934 +0000 UTC m=+6.116817533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret") pod "global-pull-secret-syncer-xn7tc" (UID: "21624337-43ef-4f45-8118-63b9de2f9ed9") : object "kube-system"/"original-pull-secret" not registered Jul 03 14:08:46.087091 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:46.087050 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-02 14:03:44 +0000 UTC" deadline="2028-01-09 17:51:38.539988373 +0000 UTC" Jul 03 14:08:46.087091 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:46.087088 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13323h42m52.452904234s" Jul 03 14:08:46.162817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:46.162782 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:08:46.162968 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:46.162926 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n9kl" podUID="514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb" Jul 03 14:08:46.164945 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:46.164787 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:46.164945 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:46.164902 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xn7tc" podUID="21624337-43ef-4f45-8118-63b9de2f9ed9" Jul 03 14:08:46.198454 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:46.198409 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-82.ec2.internal" event={"ID":"aabef1e1070cbce5457cd456994a35fb","Type":"ContainerStarted","Data":"19e2bd4c12a95f34f364acbbef2285b21282b0225dac26bbc9c6d8f36743401e"} Jul 03 14:08:46.206946 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:46.206889 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7w6pw" event={"ID":"c77a739f-276a-4602-824b-874f70e83982","Type":"ContainerStarted","Data":"1ac5747eea3d3be70ece2484efc66501c5304cab2b08e298fefd050d7a8503d1"} Jul 03 14:08:46.211395 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:46.211308 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" event={"ID":"ae42d692-7188-4fc2-b540-afebb1979245","Type":"ContainerStarted","Data":"b48507302a8cc64026df5433cc1b3fb846006970c3b256ee9cb6f5efc028e71d"} Jul 03 14:08:46.215980 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:46.214831 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-82.ec2.internal" podStartSLOduration=3.214815693 podStartE2EDuration="3.214815693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 14:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:08:46.214376425 +0000 UTC m=+4.570661072" watchObservedRunningTime="2026-07-03 14:08:46.214815693 +0000 UTC m=+4.571100315" Jul 03 14:08:46.217178 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:46.217123 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hjwhk" event={"ID":"b6e59bf8-184d-4593-aa5f-de1c5854ac77","Type":"ContainerStarted","Data":"6f2b3d1b33e3180d3703e8dba7d41891aff0ec865791c69852352c736e30776b"} Jul 03 14:08:46.220343 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:46.219740 2576 generic.go:356] "Generic (PLEG): container finished" podID="ea23e504b34f7d7760470a819fbfdb46" containerID="d084c22b7975af9acf8d0e8ccc224655f8ad48ab8cf38cb64d4ce04eb02c1eb6" exitCode=0 Jul 03 14:08:46.220343 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:46.219823 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal" event={"ID":"ea23e504b34f7d7760470a819fbfdb46","Type":"ContainerDied","Data":"d084c22b7975af9acf8d0e8ccc224655f8ad48ab8cf38cb64d4ce04eb02c1eb6"} Jul 03 14:08:46.235214 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:46.235097 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" event={"ID":"c5f6cf69-3f8b-4afe-a474-4dffa0448079","Type":"ContainerStarted","Data":"79c3bcd6b72ed20cec18ada7d2958494896e08dab4fb5e04dcb133463e411646"} Jul 03 14:08:46.243964 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:46.243932 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ksn4z" event={"ID":"988adf65-e4cb-4b10-ae2c-f689170cf998","Type":"ContainerStarted","Data":"09417d78ecd8be53d85f65ec8834dd444b779b083c5d3824b5b3ef4f78c01a1c"} Jul 03 14:08:46.260621 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:46.260576 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" event={"ID":"5515218d-73b7-4e7f-bb75-cddca2c8a06a","Type":"ContainerStarted","Data":"a901ac346e6196156b4f19f8e8780fd8ec48d835eed7f344cb3c6181a66d454f"} Jul 03 14:08:46.270115 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:46.270066 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fvf88" event={"ID":"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d","Type":"ContainerStarted","Data":"1b62d242a9025a3d26e2df7580db3ce56894a80227dae09202584d409b735d6a"} Jul 03 14:08:46.285098 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:46.285067 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lbt8h" event={"ID":"99aa3d88-a61c-4e92-adc3-a69b48070430","Type":"ContainerStarted","Data":"1ffd0cc8186ec1891b254306351c9a0e7d1790951ae8f5f0176432bfff32a05c"} Jul 03 14:08:46.301222 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:46.301142 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4v7fc" event={"ID":"928b4e88-1d06-41e9-99c9-efecc91c0ee9","Type":"ContainerStarted","Data":"268ca4a2d4a2eaea3d6933818b195c9898e6c5f26f9519184f38fef03ff42798"} Jul 03 14:08:47.162511 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:47.162477 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:08:47.163024 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:47.162614 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6n5dr" podUID="e48e7e11-89d3-4f42-be11-8f2b3c685504" Jul 03 14:08:47.346031 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:47.345997 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal" event={"ID":"ea23e504b34f7d7760470a819fbfdb46","Type":"ContainerStarted","Data":"615d37b7300225800180bfc18a784fc189e0a3cd42affe6c31529828f995e496"} Jul 03 14:08:47.677892 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:47.677854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnsvt\" (UniqueName: \"kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt\") pod \"network-check-target-6n5dr\" (UID: \"e48e7e11-89d3-4f42-be11-8f2b3c685504\") " pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:08:47.678075 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:47.678008 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 14:08:47.678075 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:47.678026 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 14:08:47.678075 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:47.678038 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cnsvt for pod openshift-network-diagnostics/network-check-target-6n5dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 14:08:47.678227 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:47.678096 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt podName:e48e7e11-89d3-4f42-be11-8f2b3c685504 nodeName:}" failed. No retries permitted until 2026-07-03 14:08:51.678079255 +0000 UTC m=+10.034363856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cnsvt" (UniqueName: "kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt") pod "network-check-target-6n5dr" (UID: "e48e7e11-89d3-4f42-be11-8f2b3c685504") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 14:08:47.779099 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:47.778812 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret\") pod \"global-pull-secret-syncer-xn7tc\" (UID: \"21624337-43ef-4f45-8118-63b9de2f9ed9\") " pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:47.779099 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:47.778882 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs\") pod \"network-metrics-daemon-7n9kl\" (UID: \"514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb\") " pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:08:47.779099 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:47.778960 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 14:08:47.779099 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:47.779002 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 14:08:47.779099 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:47.779032 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret podName:21624337-43ef-4f45-8118-63b9de2f9ed9 nodeName:}" failed. No retries permitted until 2026-07-03 14:08:51.779013101 +0000 UTC m=+10.135297701 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret") pod "global-pull-secret-syncer-xn7tc" (UID: "21624337-43ef-4f45-8118-63b9de2f9ed9") : object "kube-system"/"original-pull-secret" not registered Jul 03 14:08:47.779099 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:47.779057 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs podName:514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb nodeName:}" failed. No retries permitted until 2026-07-03 14:08:51.779041313 +0000 UTC m=+10.135325928 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs") pod "network-metrics-daemon-7n9kl" (UID: "514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 14:08:48.164859 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:48.164769 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:08:48.165305 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:48.164906 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n9kl" podUID="514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb" Jul 03 14:08:48.165546 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:48.165435 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:48.165703 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:48.165539 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xn7tc" podUID="21624337-43ef-4f45-8118-63b9de2f9ed9" Jul 03 14:08:49.163054 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:49.162535 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:08:49.163054 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:49.162690 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6n5dr" podUID="e48e7e11-89d3-4f42-be11-8f2b3c685504" Jul 03 14:08:50.162702 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:50.162647 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:08:50.163113 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:50.162813 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n9kl" podUID="514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb" Jul 03 14:08:50.163174 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:50.162679 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:50.163283 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:50.163262 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xn7tc" podUID="21624337-43ef-4f45-8118-63b9de2f9ed9" Jul 03 14:08:51.161955 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:51.161921 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:08:51.162154 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:51.162042 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6n5dr" podUID="e48e7e11-89d3-4f42-be11-8f2b3c685504" Jul 03 14:08:51.715275 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:51.715219 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnsvt\" (UniqueName: \"kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt\") pod \"network-check-target-6n5dr\" (UID: \"e48e7e11-89d3-4f42-be11-8f2b3c685504\") " pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:08:51.715761 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:51.715503 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 14:08:51.715761 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:51.715526 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 14:08:51.715761 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:51.715538 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cnsvt for pod openshift-network-diagnostics/network-check-target-6n5dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 14:08:51.715761 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:51.715601 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt podName:e48e7e11-89d3-4f42-be11-8f2b3c685504 nodeName:}" failed. No retries permitted until 2026-07-03 14:08:59.715581686 +0000 UTC m=+18.071866301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cnsvt" (UniqueName: "kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt") pod "network-check-target-6n5dr" (UID: "e48e7e11-89d3-4f42-be11-8f2b3c685504") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 14:08:51.815871 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:51.815830 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret\") pod \"global-pull-secret-syncer-xn7tc\" (UID: \"21624337-43ef-4f45-8118-63b9de2f9ed9\") " pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:51.816049 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:51.815891 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs\") pod \"network-metrics-daemon-7n9kl\" (UID: \"514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb\") " pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:08:51.816049 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:51.815987 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 14:08:51.816049 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:51.815988 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 14:08:51.816172 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:51.816058 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret podName:21624337-43ef-4f45-8118-63b9de2f9ed9 nodeName:}" failed. No retries permitted until 2026-07-03 14:08:59.816038378 +0000 UTC m=+18.172322999 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret") pod "global-pull-secret-syncer-xn7tc" (UID: "21624337-43ef-4f45-8118-63b9de2f9ed9") : object "kube-system"/"original-pull-secret" not registered Jul 03 14:08:51.816172 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:51.816088 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs podName:514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb nodeName:}" failed. No retries permitted until 2026-07-03 14:08:59.816074572 +0000 UTC m=+18.172359173 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs") pod "network-metrics-daemon-7n9kl" (UID: "514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 14:08:52.162669 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:52.162618 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:08:52.162885 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:52.162771 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n9kl" podUID="514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb" Jul 03 14:08:52.163288 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:52.163258 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:52.163401 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:52.163362 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xn7tc" podUID="21624337-43ef-4f45-8118-63b9de2f9ed9" Jul 03 14:08:53.162329 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:53.162291 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:08:53.162821 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:53.162439 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6n5dr" podUID="e48e7e11-89d3-4f42-be11-8f2b3c685504" Jul 03 14:08:54.162041 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:54.162000 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:08:54.162041 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:54.162036 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:54.162284 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:54.162168 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n9kl" podUID="514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb" Jul 03 14:08:54.162340 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:54.162274 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xn7tc" podUID="21624337-43ef-4f45-8118-63b9de2f9ed9" Jul 03 14:08:55.162227 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:55.161824 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:08:55.162227 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:55.161948 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6n5dr" podUID="e48e7e11-89d3-4f42-be11-8f2b3c685504" Jul 03 14:08:56.162269 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:56.162230 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:08:56.162758 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:56.162230 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:56.162758 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:56.162364 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n9kl" podUID="514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb" Jul 03 14:08:56.162758 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:56.162467 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xn7tc" podUID="21624337-43ef-4f45-8118-63b9de2f9ed9" Jul 03 14:08:57.162563 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:57.162536 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:08:57.162992 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:57.162642 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6n5dr" podUID="e48e7e11-89d3-4f42-be11-8f2b3c685504" Jul 03 14:08:58.162199 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:58.162166 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:58.162382 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:58.162301 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xn7tc" podUID="21624337-43ef-4f45-8118-63b9de2f9ed9" Jul 03 14:08:58.162382 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:58.162344 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:08:58.162509 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:58.162430 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n9kl" podUID="514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb" Jul 03 14:08:59.162518 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:59.162473 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:08:59.163029 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:59.162610 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6n5dr" podUID="e48e7e11-89d3-4f42-be11-8f2b3c685504" Jul 03 14:08:59.775476 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:59.775432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnsvt\" (UniqueName: \"kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt\") pod \"network-check-target-6n5dr\" (UID: \"e48e7e11-89d3-4f42-be11-8f2b3c685504\") " pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:08:59.775672 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:59.775611 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 14:08:59.775672 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:59.775635 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 14:08:59.775672 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:59.775645 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cnsvt for pod openshift-network-diagnostics/network-check-target-6n5dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 14:08:59.775821 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:59.775718 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt podName:e48e7e11-89d3-4f42-be11-8f2b3c685504 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:15.775697769 +0000 UTC m=+34.131982367 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cnsvt" (UniqueName: "kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt") pod "network-check-target-6n5dr" (UID: "e48e7e11-89d3-4f42-be11-8f2b3c685504") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 14:08:59.876057 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:59.876012 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret\") pod \"global-pull-secret-syncer-xn7tc\" (UID: \"21624337-43ef-4f45-8118-63b9de2f9ed9\") " pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:08:59.876222 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:08:59.876070 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs\") pod \"network-metrics-daemon-7n9kl\" (UID: \"514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb\") " pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:08:59.876222 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:59.876176 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 14:08:59.876222 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:59.876176 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 14:08:59.876318 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:59.876242 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret podName:21624337-43ef-4f45-8118-63b9de2f9ed9 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:15.876226611 +0000 UTC m=+34.232511232 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret") pod "global-pull-secret-syncer-xn7tc" (UID: "21624337-43ef-4f45-8118-63b9de2f9ed9") : object "kube-system"/"original-pull-secret" not registered Jul 03 14:08:59.876318 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:08:59.876258 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs podName:514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb nodeName:}" failed. No retries permitted until 2026-07-03 14:09:15.87625199 +0000 UTC m=+34.232536588 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs") pod "network-metrics-daemon-7n9kl" (UID: "514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 14:09:00.162253 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:00.162223 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:09:00.162458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:00.162224 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:09:00.162458 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:00.162344 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xn7tc" podUID="21624337-43ef-4f45-8118-63b9de2f9ed9" Jul 03 14:09:00.162561 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:00.162457 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n9kl" podUID="514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb" Jul 03 14:09:01.162669 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:01.162622 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:09:01.163067 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:01.162803 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6n5dr" podUID="e48e7e11-89d3-4f42-be11-8f2b3c685504" Jul 03 14:09:02.164472 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:02.164428 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:09:02.164954 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:02.164592 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n9kl" podUID="514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb" Jul 03 14:09:02.166705 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:02.165069 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:09:02.166705 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:02.165193 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xn7tc" podUID="21624337-43ef-4f45-8118-63b9de2f9ed9" Jul 03 14:09:03.162687 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:03.162636 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:09:03.162979 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:03.162764 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6n5dr" podUID="e48e7e11-89d3-4f42-be11-8f2b3c685504" Jul 03 14:09:04.164293 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:04.164265 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:09:04.164724 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:04.164439 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n9kl" podUID="514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb" Jul 03 14:09:04.164789 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:04.164741 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:09:04.164844 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:04.164824 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xn7tc" podUID="21624337-43ef-4f45-8118-63b9de2f9ed9" Jul 03 14:09:04.379714 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:04.379499 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" event={"ID":"5515218d-73b7-4e7f-bb75-cddca2c8a06a","Type":"ContainerStarted","Data":"037a616420c9582f16b3ba69cd5073275989c83c35c59fd83a21bf39d5685836"} Jul 03 14:09:04.380807 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:04.380786 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fvf88" event={"ID":"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d","Type":"ContainerStarted","Data":"dc4765b1d1803b0030a8fc0f0685c34f631acc53fb55d190004b9e186295d574"} Jul 03 14:09:04.382925 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:04.382903 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lbt8h" event={"ID":"99aa3d88-a61c-4e92-adc3-a69b48070430","Type":"ContainerStarted","Data":"2ca2d19c8d58beadbed3533795cf6913e34fad305e126c7822c007441085c0de"} Jul 03 14:09:04.384204 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:04.384186 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4v7fc" event={"ID":"928b4e88-1d06-41e9-99c9-efecc91c0ee9","Type":"ContainerStarted","Data":"36639250f97a03247c625dff68d7c8391d17fb85faad6d941ea4022ab0424c26"} Jul 03 14:09:04.411502 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:04.410516 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-82.ec2.internal" podStartSLOduration=21.410495622 podStartE2EDuration="21.410495622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 14:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:08:47.37088299 +0000 UTC m=+5.727167612" watchObservedRunningTime="2026-07-03 14:09:04.410495622 +0000 UTC m=+22.766780244" Jul 03 14:09:04.433310 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:04.433266 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4v7fc" podStartSLOduration=3.570243547 podStartE2EDuration="22.433245393s" totalImagesPullingTime="18.863001846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 14:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:04.43273924 +0000 UTC m=+22.789023861" watchObservedRunningTime="2026-07-03 14:09:04.433245393 +0000 UTC m=+22.789530003" Jul 03 14:09:04.458519 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:04.458480 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lbt8h" podStartSLOduration=7.810088808 podStartE2EDuration="22.458464489s" totalImagesPullingTime="14.648375681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:04.4583253 +0000 UTC m=+22.814609921" watchObservedRunningTime="2026-07-03 14:09:04.458464489 +0000 UTC m=+22.814749109" Jul 03 14:09:05.162156 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:05.161990 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:09:05.162315 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:05.162268 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6n5dr" podUID="e48e7e11-89d3-4f42-be11-8f2b3c685504" Jul 03 14:09:05.388426 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:05.388396 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7w6pw" event={"ID":"c77a739f-276a-4602-824b-874f70e83982","Type":"ContainerStarted","Data":"a48c141f49f266ef40a5636937614d3def0742314d86a05b2f2101047033c22b"} Jul 03 14:09:05.389960 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:05.389935 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" event={"ID":"ae42d692-7188-4fc2-b540-afebb1979245","Type":"ContainerStarted","Data":"d0add8cbddc7a5a7f6839440aa36f080f94691a0a8c3e13a4e364433e42eaa45"} Jul 03 14:09:05.392984 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:05.392954 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hjwhk" event={"ID":"b6e59bf8-184d-4593-aa5f-de1c5854ac77","Type":"ContainerStarted","Data":"8bec4056cf4b60becf2462d02a1aaf038fbf31f864b9b450e8c57c732cb1c168"} Jul 03 14:09:05.395354 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:05.395338 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:09:05.395607 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:05.395583 2576 generic.go:356] "Generic (PLEG): container finished" podID="c5f6cf69-3f8b-4afe-a474-4dffa0448079" containerID="1eaf7d32943b4df8cf422522484ee4f54b2bf116395bd96a7abd5477e6ee91b6" exitCode=1 Jul 03 14:09:05.395688 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:05.395613 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" event={"ID":"c5f6cf69-3f8b-4afe-a474-4dffa0448079","Type":"ContainerStarted","Data":"fc521e6f59b762c4ef0f9d73860905b8426d2af07d10ea96b9a84c418e442c52"} Jul 03 14:09:05.395743 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:05.395696 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" event={"ID":"c5f6cf69-3f8b-4afe-a474-4dffa0448079","Type":"ContainerStarted","Data":"341c85e7f3fd576027a12442f8e0e85be06439d23ff6f17b5c22caad331c3ab4"} Jul 03 14:09:05.395743 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:05.395736 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" event={"ID":"c5f6cf69-3f8b-4afe-a474-4dffa0448079","Type":"ContainerStarted","Data":"9f634285ea764868405ff322a8e37814038cda4491dd87750d9ce4c63c1b4be6"} Jul 03 14:09:05.395822 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:05.395775 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" event={"ID":"c5f6cf69-3f8b-4afe-a474-4dffa0448079","Type":"ContainerStarted","Data":"0be3ed645db98a1228bbb3d894a270f8d694ff67e35d3bfbe93601a9ae1e52a6"} Jul 03 14:09:05.395822 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:05.395805 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" event={"ID":"c5f6cf69-3f8b-4afe-a474-4dffa0448079","Type":"ContainerDied","Data":"1eaf7d32943b4df8cf422522484ee4f54b2bf116395bd96a7abd5477e6ee91b6"} Jul 03 14:09:05.395896 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:05.395842 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" event={"ID":"c5f6cf69-3f8b-4afe-a474-4dffa0448079","Type":"ContainerStarted","Data":"e511f883c5af37874bcc2be083e1a18717dcdc26e7e03d5a40eb8ac643082971"} Jul 03 14:09:05.396813 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:05.396797 2576 generic.go:356] "Generic (PLEG): container finished" podID="f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d" containerID="dc4765b1d1803b0030a8fc0f0685c34f631acc53fb55d190004b9e186295d574" exitCode=0 Jul 03 14:09:05.396885 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:05.396868 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fvf88" event={"ID":"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d","Type":"ContainerDied","Data":"dc4765b1d1803b0030a8fc0f0685c34f631acc53fb55d190004b9e186295d574"} Jul 03 14:09:05.411339 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:05.411304 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7w6pw" podStartSLOduration=4.512484234 podStartE2EDuration="23.411291069s" totalImagesPullingTime="18.898806835s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 14:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:05.410766715 +0000 UTC m=+23.767051338" watchObservedRunningTime="2026-07-03 14:09:05.411291069 +0000 UTC m=+23.767575689" Jul 03 14:09:05.426567 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:05.426544 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 03 14:09:05.465201 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:05.465153 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v2kxm" podStartSLOduration=4.61450167 podStartE2EDuration="23.465139763s" totalImagesPullingTime="18.850638093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 14:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:05.464791831 +0000 UTC m=+23.821076450" watchObservedRunningTime="2026-07-03 14:09:05.465139763 +0000 UTC m=+23.821424382" Jul 03 14:09:05.482105 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:05.482048 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hjwhk" podStartSLOduration=4.635950681 podStartE2EDuration="23.482031785s" totalImagesPullingTime="18.846081104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 14:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:05.48188961 +0000 UTC m=+23.838174230" watchObservedRunningTime="2026-07-03 14:09:05.482031785 +0000 UTC m=+23.838316405" Jul 03 14:09:06.101122 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:06.100912 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-03T14:09:05.4265641Z","UUID":"baa912a2-bac8-4323-889f-4c967fed68fa","Handler":null,"Name":"","Endpoint":""} Jul 03 14:09:06.103106 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:06.103084 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 03 14:09:06.103260 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:06.103130 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 03 14:09:06.162423 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:06.162362 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:09:06.162423 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:06.162390 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:09:06.162738 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:06.162523 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n9kl" podUID="514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb" Jul 03 14:09:06.162738 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:06.162649 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xn7tc" podUID="21624337-43ef-4f45-8118-63b9de2f9ed9" Jul 03 14:09:06.400284 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:06.400195 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ksn4z" event={"ID":"988adf65-e4cb-4b10-ae2c-f689170cf998","Type":"ContainerStarted","Data":"c593415c13ef5fffd68c8688ecfeeb2d0c396bcb6128c97152a46329df82940a"} Jul 03 14:09:06.402251 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:06.402227 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" event={"ID":"5515218d-73b7-4e7f-bb75-cddca2c8a06a","Type":"ContainerStarted","Data":"00dcb4092c3c417ee9d1a3e763833e34c4f4316790c37d5b754ae09f5a3dbabd"} Jul 03 14:09:06.402367 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:06.402269 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" event={"ID":"5515218d-73b7-4e7f-bb75-cddca2c8a06a","Type":"ContainerStarted","Data":"1857899f73f6c7734bacb18a1df0eaa4300d9690863bbe7d2ab244073cc17d3d"} Jul 03 14:09:06.446182 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:06.446116 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ksn4z" podStartSLOduration=5.607449248 podStartE2EDuration="24.446096056s" totalImagesPullingTime="18.838646808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 14:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:06.420008236 +0000 UTC m=+24.776292856" watchObservedRunningTime="2026-07-03 14:09:06.446096056 +0000 UTC m=+24.802380676" Jul 03 14:09:06.446374 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:06.446347 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lflw5" podStartSLOduration=3.797053329 podStartE2EDuration="24.446339584s" totalImagesPullingTime="20.649286255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 14:08:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:06.445879136 +0000 UTC m=+24.802163748" watchObservedRunningTime="2026-07-03 14:09:06.446339584 +0000 UTC m=+24.802624206" Jul 03 14:09:07.162196 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:07.162115 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:09:07.162350 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:07.162211 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6n5dr" podUID="e48e7e11-89d3-4f42-be11-8f2b3c685504" Jul 03 14:09:07.406839 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:07.406812 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:09:07.407212 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:07.407184 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" event={"ID":"c5f6cf69-3f8b-4afe-a474-4dffa0448079","Type":"ContainerStarted","Data":"75989e291b47aad557e47bf22d0eb60eddc9bbc557b4352a3b77866f45b4c435"} Jul 03 14:09:08.162085 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:08.161895 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:09:08.162299 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:08.161895 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:09:08.162299 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:08.162187 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xn7tc" podUID="21624337-43ef-4f45-8118-63b9de2f9ed9" Jul 03 14:09:08.162299 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:08.162271 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n9kl" podUID="514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb" Jul 03 14:09:08.225455 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:08.225423 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lbt8h" Jul 03 14:09:08.226102 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:08.226085 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lbt8h" Jul 03 14:09:08.501594 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:08.501499 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lbt8h" Jul 03 14:09:08.502208 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:08.502188 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lbt8h" Jul 03 14:09:09.162466 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:09.162339 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:09:09.162613 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:09.162466 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6n5dr" podUID="e48e7e11-89d3-4f42-be11-8f2b3c685504" Jul 03 14:09:09.415242 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:09.415165 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:09:09.415599 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:09.415567 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" event={"ID":"c5f6cf69-3f8b-4afe-a474-4dffa0448079","Type":"ContainerStarted","Data":"c6d09c027b22bd444dd73667cd89f202dec95a4bb3f5ee8c46ccf5be8b45f5b0"} Jul 03 14:09:09.416222 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:09.416200 2576 scope.go:117] "RemoveContainer" containerID="1eaf7d32943b4df8cf422522484ee4f54b2bf116395bd96a7abd5477e6ee91b6" Jul 03 14:09:10.049749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:10.049477 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:09:10.162915 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:10.162351 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:09:10.162915 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:10.162502 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xn7tc" podUID="21624337-43ef-4f45-8118-63b9de2f9ed9" Jul 03 14:09:10.162915 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:10.162829 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:09:10.163205 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:10.162935 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n9kl" podUID="514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb" Jul 03 14:09:11.162552 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:11.162514 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:09:11.162983 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:11.162630 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6n5dr" podUID="e48e7e11-89d3-4f42-be11-8f2b3c685504" Jul 03 14:09:11.179789 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:11.179758 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xn7tc"] Jul 03 14:09:11.179954 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:11.179870 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:09:11.180011 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:11.179983 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xn7tc" podUID="21624337-43ef-4f45-8118-63b9de2f9ed9" Jul 03 14:09:11.180283 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:11.180260 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7n9kl"] Jul 03 14:09:11.180355 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:11.180340 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:09:11.180426 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:11.180410 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n9kl" podUID="514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb" Jul 03 14:09:11.182647 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:11.182625 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6n5dr"] Jul 03 14:09:11.424696 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:11.424648 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:09:11.425087 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:11.425059 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" event={"ID":"c5f6cf69-3f8b-4afe-a474-4dffa0448079","Type":"ContainerStarted","Data":"68df854e59b135fd7cd89ae1e45e8629f4ecaaa1890bf803ee76cecccae33229"} Jul 03 14:09:11.425268 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:11.425239 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:09:11.425365 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:11.425352 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:09:11.427388 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:11.427365 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:09:11.427475 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:11.427402 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fvf88" event={"ID":"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d","Type":"ContainerStarted","Data":"e18dc51b7d7593666843dda5f20aaeb6bdb70c43c260456aeb9facb33d92c4c3"} Jul 03 14:09:11.427513 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:11.427492 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6n5dr" podUID="e48e7e11-89d3-4f42-be11-8f2b3c685504" Jul 03 14:09:11.441808 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:11.441782 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:09:11.441973 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:11.441919 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:09:11.458266 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:11.458223 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" podStartSLOduration=10.567386049 podStartE2EDuration="29.458209216s" totalImagesPullingTime="18.890823167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 14:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:11.456571938 +0000 UTC m=+29.812856558" watchObservedRunningTime="2026-07-03 14:09:11.458209216 +0000 UTC m=+29.814493836" Jul 03 14:09:12.431559 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:12.431366 2576 generic.go:356] "Generic (PLEG): container finished" podID="f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d" containerID="e18dc51b7d7593666843dda5f20aaeb6bdb70c43c260456aeb9facb33d92c4c3" exitCode=0 Jul 03 14:09:12.431559 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:12.431428 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fvf88" event={"ID":"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d","Type":"ContainerDied","Data":"e18dc51b7d7593666843dda5f20aaeb6bdb70c43c260456aeb9facb33d92c4c3"} Jul 03 14:09:13.162329 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:13.162297 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:09:13.162329 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:13.162323 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:09:13.162572 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:13.162323 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:09:13.162572 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:13.162418 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xn7tc" podUID="21624337-43ef-4f45-8118-63b9de2f9ed9" Jul 03 14:09:13.162572 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:13.162503 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n9kl" podUID="514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb" Jul 03 14:09:13.162737 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:13.162573 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6n5dr" podUID="e48e7e11-89d3-4f42-be11-8f2b3c685504" Jul 03 14:09:13.450727 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:13.450680 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" podUID="c5f6cf69-3f8b-4afe-a474-4dffa0448079" containerName="ovnkube-controller" probeResult="failure" output="" Jul 03 14:09:14.438918 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:14.438887 2576 generic.go:356] "Generic (PLEG): container finished" podID="f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d" containerID="e545c681057e00e18c7d67fb2acafc879c1f2fc03a9703992783aea5f5f9214d" exitCode=0 Jul 03 14:09:14.439087 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:14.438935 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fvf88" event={"ID":"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d","Type":"ContainerDied","Data":"e545c681057e00e18c7d67fb2acafc879c1f2fc03a9703992783aea5f5f9214d"} Jul 03 14:09:15.162388 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:15.162314 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:09:15.162388 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:15.162360 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:09:15.163114 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:15.162416 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:09:15.163114 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:15.162425 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xn7tc" podUID="21624337-43ef-4f45-8118-63b9de2f9ed9" Jul 03 14:09:15.163114 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:15.162510 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6n5dr" podUID="e48e7e11-89d3-4f42-be11-8f2b3c685504" Jul 03 14:09:15.163114 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:15.162581 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n9kl" podUID="514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb" Jul 03 14:09:15.793513 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:15.793484 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnsvt\" (UniqueName: \"kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt\") pod \"network-check-target-6n5dr\" (UID: \"e48e7e11-89d3-4f42-be11-8f2b3c685504\") " pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:09:15.793744 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:15.793630 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 14:09:15.793744 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:15.793650 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 14:09:15.793744 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:15.793678 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cnsvt for pod openshift-network-diagnostics/network-check-target-6n5dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 14:09:15.793744 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:15.793728 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt podName:e48e7e11-89d3-4f42-be11-8f2b3c685504 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:47.793712203 +0000 UTC m=+66.149996822 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cnsvt" (UniqueName: "kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt") pod "network-check-target-6n5dr" (UID: "e48e7e11-89d3-4f42-be11-8f2b3c685504") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 14:09:15.894740 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:15.894710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret\") pod \"global-pull-secret-syncer-xn7tc\" (UID: \"21624337-43ef-4f45-8118-63b9de2f9ed9\") " pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:09:15.894955 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:15.894752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs\") pod \"network-metrics-daemon-7n9kl\" (UID: \"514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb\") " pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:09:15.894955 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:15.894837 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 14:09:15.894955 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:15.894840 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 14:09:15.894955 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:15.894884 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs podName:514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb nodeName:}" failed. No retries permitted until 2026-07-03 14:09:47.894870828 +0000 UTC m=+66.251155426 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs") pod "network-metrics-daemon-7n9kl" (UID: "514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 14:09:15.894955 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:15.894896 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret podName:21624337-43ef-4f45-8118-63b9de2f9ed9 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:47.894890799 +0000 UTC m=+66.251175397 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret") pod "global-pull-secret-syncer-xn7tc" (UID: "21624337-43ef-4f45-8118-63b9de2f9ed9") : object "kube-system"/"original-pull-secret" not registered Jul 03 14:09:16.446027 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:16.445980 2576 generic.go:356] "Generic (PLEG): container finished" podID="f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d" containerID="9488e6991a2a37959b8898eb3047c04e76687f542ec611888dd7ffb040d23de2" exitCode=0 Jul 03 14:09:16.446496 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:16.446037 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fvf88" event={"ID":"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d","Type":"ContainerDied","Data":"9488e6991a2a37959b8898eb3047c04e76687f542ec611888dd7ffb040d23de2"} Jul 03 14:09:16.900412 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:16.900373 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-82.ec2.internal" event="NodeReady" Jul 03 14:09:16.900591 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:16.900506 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 03 14:09:16.950938 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:16.950904 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wx4zj"] Jul 03 14:09:16.969324 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:16.969251 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tn4pv"] Jul 03 14:09:16.969479 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:16.969410 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:16.973512 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:16.973488 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 03 14:09:16.973879 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:16.973525 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wf6pg\"" Jul 03 14:09:16.973879 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:16.973849 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 03 14:09:16.983285 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:16.983133 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wx4zj"] Jul 03 14:09:16.983385 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:16.983303 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tn4pv"] Jul 03 14:09:16.983385 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:16.983238 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tn4pv" Jul 03 14:09:16.985952 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:16.985931 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 03 14:09:16.986061 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:16.986022 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 03 14:09:16.986146 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:16.986131 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 03 14:09:16.986281 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:16.986269 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7fg4h\"" Jul 03 14:09:17.101100 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.101066 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls\") pod \"dns-default-wx4zj\" (UID: \"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4\") " pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:17.101298 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.101112 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-tmp-dir\") pod \"dns-default-wx4zj\" (UID: \"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4\") " pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:17.101298 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.101171 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-config-volume\") pod \"dns-default-wx4zj\" (UID: \"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4\") " pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:17.101298 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.101188 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwf2r\" (UniqueName: \"kubernetes.io/projected/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-kube-api-access-jwf2r\") pod \"dns-default-wx4zj\" (UID: \"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4\") " pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:17.101298 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.101215 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68e8c775-86ec-4944-9379-70d1bcfdbf69-cert\") pod \"ingress-canary-tn4pv\" (UID: \"68e8c775-86ec-4944-9379-70d1bcfdbf69\") " pod="openshift-ingress-canary/ingress-canary-tn4pv" Jul 03 14:09:17.101298 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.101291 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dhqj\" (UniqueName: \"kubernetes.io/projected/68e8c775-86ec-4944-9379-70d1bcfdbf69-kube-api-access-9dhqj\") pod \"ingress-canary-tn4pv\" (UID: \"68e8c775-86ec-4944-9379-70d1bcfdbf69\") " pod="openshift-ingress-canary/ingress-canary-tn4pv" Jul 03 14:09:17.162720 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.162678 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:09:17.162720 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.162709 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:09:17.163035 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.163005 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:09:17.165982 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.165959 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 03 14:09:17.166508 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.166422 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 03 14:09:17.166508 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.166454 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 03 14:09:17.166711 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.166547 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-472cc\"" Jul 03 14:09:17.166711 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.166708 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tlbvf\"" Jul 03 14:09:17.167362 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.167286 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 03 14:09:17.202370 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.202344 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dhqj\" (UniqueName: \"kubernetes.io/projected/68e8c775-86ec-4944-9379-70d1bcfdbf69-kube-api-access-9dhqj\") pod \"ingress-canary-tn4pv\" (UID: \"68e8c775-86ec-4944-9379-70d1bcfdbf69\") " pod="openshift-ingress-canary/ingress-canary-tn4pv" Jul 03 14:09:17.202533 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.202382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls\") pod \"dns-default-wx4zj\" (UID: \"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4\") " pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:17.202533 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.202414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-tmp-dir\") pod \"dns-default-wx4zj\" (UID: \"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4\") " pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:17.202533 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.202465 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-config-volume\") pod \"dns-default-wx4zj\" (UID: \"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4\") " pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:17.202533 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.202484 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwf2r\" (UniqueName: \"kubernetes.io/projected/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-kube-api-access-jwf2r\") pod \"dns-default-wx4zj\" (UID: \"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4\") " pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:17.202533 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.202511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68e8c775-86ec-4944-9379-70d1bcfdbf69-cert\") pod \"ingress-canary-tn4pv\" (UID: \"68e8c775-86ec-4944-9379-70d1bcfdbf69\") " pod="openshift-ingress-canary/ingress-canary-tn4pv" Jul 03 14:09:17.202959 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.202940 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-tmp-dir\") pod \"dns-default-wx4zj\" (UID: \"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4\") " pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:17.203057 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:17.203039 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 14:09:17.203117 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:17.203113 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls podName:2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:17.703094638 +0000 UTC m=+36.059379236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls") pod "dns-default-wx4zj" (UID: "2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4") : secret "dns-default-metrics-tls" not found Jul 03 14:09:17.203438 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.203414 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-config-volume\") pod \"dns-default-wx4zj\" (UID: \"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4\") " pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:17.207763 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.207741 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68e8c775-86ec-4944-9379-70d1bcfdbf69-cert\") pod \"ingress-canary-tn4pv\" (UID: \"68e8c775-86ec-4944-9379-70d1bcfdbf69\") " pod="openshift-ingress-canary/ingress-canary-tn4pv" Jul 03 14:09:17.215400 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.215362 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwf2r\" (UniqueName: \"kubernetes.io/projected/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-kube-api-access-jwf2r\") pod \"dns-default-wx4zj\" (UID: \"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4\") " pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:17.215400 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.215378 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dhqj\" (UniqueName: \"kubernetes.io/projected/68e8c775-86ec-4944-9379-70d1bcfdbf69-kube-api-access-9dhqj\") pod \"ingress-canary-tn4pv\" (UID: \"68e8c775-86ec-4944-9379-70d1bcfdbf69\") " pod="openshift-ingress-canary/ingress-canary-tn4pv" Jul 03 14:09:17.291473 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.291435 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tn4pv" Jul 03 14:09:17.499013 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.498981 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tn4pv"] Jul 03 14:09:17.503373 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:09:17.503338 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e8c775_86ec_4944_9379_70d1bcfdbf69.slice/crio-5644f502dcf60936c19a8d4de0ca6d2d050d67f9654cd17eb34767812d0396a6 WatchSource:0}: Error finding container 5644f502dcf60936c19a8d4de0ca6d2d050d67f9654cd17eb34767812d0396a6: Status 404 returned error can't find the container with id 5644f502dcf60936c19a8d4de0ca6d2d050d67f9654cd17eb34767812d0396a6 Jul 03 14:09:17.706888 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:17.706806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls\") pod \"dns-default-wx4zj\" (UID: \"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4\") " pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:17.707045 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:17.706934 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 14:09:17.707045 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:17.706986 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls podName:2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:18.70697125 +0000 UTC m=+37.063255849 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls") pod "dns-default-wx4zj" (UID: "2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4") : secret "dns-default-metrics-tls" not found Jul 03 14:09:18.042507 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.042473 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw"] Jul 03 14:09:18.055759 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.055727 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-6fj84"] Jul 03 14:09:18.055927 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.055841 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw" Jul 03 14:09:18.058729 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.058697 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 03 14:09:18.058946 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.058932 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-k6m5z\"" Jul 03 14:09:18.070429 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.070231 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 03 14:09:18.070429 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.070336 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 03 14:09:18.074923 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.074899 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-578dddfc57-dnhg6"] Jul 03 14:09:18.075057 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.075030 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-6fj84" Jul 03 14:09:18.082031 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.082008 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xchwz\"" Jul 03 14:09:18.082204 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.082188 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 03 14:09:18.082386 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.082364 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 03 14:09:18.082919 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.082901 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 03 14:09:18.083448 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.083268 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 03 14:09:18.089401 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.089381 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 03 14:09:18.099075 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.099052 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw"] Jul 03 14:09:18.099212 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.099099 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-6fj84"] Jul 03 14:09:18.099212 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.099124 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-578dddfc57-dnhg6"] Jul 03 14:09:18.099212 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.099171 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:18.103805 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.103781 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 03 14:09:18.104077 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.104061 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 03 14:09:18.104316 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.104300 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 03 14:09:18.104542 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.104523 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 03 14:09:18.104625 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.104607 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 03 14:09:18.104784 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.104766 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 03 14:09:18.104846 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.104799 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cvjmf\"" Jul 03 14:09:18.210352 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.210316 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/baef0a27-3024-4690-bb09-fee580be8129-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-j8nfw\" (UID: \"baef0a27-3024-4690-bb09-fee580be8129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw" Jul 03 14:09:18.210545 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.210375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-stats-auth\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:18.210545 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.210406 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:18.210545 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.210442 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cdde4f72-8103-49d7-b324-366692ef7c99-trusted-ca\") pod \"console-operator-68b757fb67-6fj84\" (UID: \"cdde4f72-8103-49d7-b324-366692ef7c99\") " pod="openshift-console-operator/console-operator-68b757fb67-6fj84" Jul 03 14:09:18.210545 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.210481 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdde4f72-8103-49d7-b324-366692ef7c99-serving-cert\") pod \"console-operator-68b757fb67-6fj84\" (UID: \"cdde4f72-8103-49d7-b324-366692ef7c99\") " pod="openshift-console-operator/console-operator-68b757fb67-6fj84" Jul 03 14:09:18.210545 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.210516 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8bt5\" (UniqueName: \"kubernetes.io/projected/baef0a27-3024-4690-bb09-fee580be8129-kube-api-access-r8bt5\") pod \"cluster-samples-operator-56fcf86bf-j8nfw\" (UID: \"baef0a27-3024-4690-bb09-fee580be8129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw" Jul 03 14:09:18.210545 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.210541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smwkl\" (UniqueName: \"kubernetes.io/projected/cdde4f72-8103-49d7-b324-366692ef7c99-kube-api-access-smwkl\") pod \"console-operator-68b757fb67-6fj84\" (UID: \"cdde4f72-8103-49d7-b324-366692ef7c99\") " pod="openshift-console-operator/console-operator-68b757fb67-6fj84" Jul 03 14:09:18.210876 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.210580 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdde4f72-8103-49d7-b324-366692ef7c99-config\") pod \"console-operator-68b757fb67-6fj84\" (UID: \"cdde4f72-8103-49d7-b324-366692ef7c99\") " pod="openshift-console-operator/console-operator-68b757fb67-6fj84" Jul 03 14:09:18.210876 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.210632 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:18.210876 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.210683 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ztn9\" (UniqueName: \"kubernetes.io/projected/18604274-740f-4f0b-bc1c-fb45e29fdc12-kube-api-access-8ztn9\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:18.210876 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.210716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-default-certificate\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:18.311923 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.311835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/baef0a27-3024-4690-bb09-fee580be8129-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-j8nfw\" (UID: \"baef0a27-3024-4690-bb09-fee580be8129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw" Jul 03 14:09:18.311923 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.311898 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-stats-auth\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:18.311923 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.311921 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:18.312191 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.311942 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cdde4f72-8103-49d7-b324-366692ef7c99-trusted-ca\") pod \"console-operator-68b757fb67-6fj84\" (UID: \"cdde4f72-8103-49d7-b324-366692ef7c99\") " pod="openshift-console-operator/console-operator-68b757fb67-6fj84" Jul 03 14:09:18.312191 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.311970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdde4f72-8103-49d7-b324-366692ef7c99-serving-cert\") pod \"console-operator-68b757fb67-6fj84\" (UID: \"cdde4f72-8103-49d7-b324-366692ef7c99\") " pod="openshift-console-operator/console-operator-68b757fb67-6fj84" Jul 03 14:09:18.312191 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.311995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8bt5\" (UniqueName: \"kubernetes.io/projected/baef0a27-3024-4690-bb09-fee580be8129-kube-api-access-r8bt5\") pod \"cluster-samples-operator-56fcf86bf-j8nfw\" (UID: \"baef0a27-3024-4690-bb09-fee580be8129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw" Jul 03 14:09:18.312191 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:18.312005 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 03 14:09:18.312191 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:18.312081 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baef0a27-3024-4690-bb09-fee580be8129-samples-operator-tls podName:baef0a27-3024-4690-bb09-fee580be8129 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:18.812059616 +0000 UTC m=+37.168344233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/baef0a27-3024-4690-bb09-fee580be8129-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-j8nfw" (UID: "baef0a27-3024-4690-bb09-fee580be8129") : secret "samples-operator-tls" not found Jul 03 14:09:18.312438 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.312013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smwkl\" (UniqueName: \"kubernetes.io/projected/cdde4f72-8103-49d7-b324-366692ef7c99-kube-api-access-smwkl\") pod \"console-operator-68b757fb67-6fj84\" (UID: \"cdde4f72-8103-49d7-b324-366692ef7c99\") " pod="openshift-console-operator/console-operator-68b757fb67-6fj84" Jul 03 14:09:18.312438 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.312409 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdde4f72-8103-49d7-b324-366692ef7c99-config\") pod \"console-operator-68b757fb67-6fj84\" (UID: \"cdde4f72-8103-49d7-b324-366692ef7c99\") " pod="openshift-console-operator/console-operator-68b757fb67-6fj84" Jul 03 14:09:18.312541 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.312465 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:18.312541 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.312495 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ztn9\" (UniqueName: \"kubernetes.io/projected/18604274-740f-4f0b-bc1c-fb45e29fdc12-kube-api-access-8ztn9\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:18.312633 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.312554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-default-certificate\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:18.312735 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:18.312719 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle podName:18604274-740f-4f0b-bc1c-fb45e29fdc12 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:18.81270467 +0000 UTC m=+37.168989299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle") pod "router-default-578dddfc57-dnhg6" (UID: "18604274-740f-4f0b-bc1c-fb45e29fdc12") : configmap references non-existent config key: service-ca.crt Jul 03 14:09:18.312813 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:18.312797 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 03 14:09:18.312865 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:18.312837 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs podName:18604274-740f-4f0b-bc1c-fb45e29fdc12 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:18.812824899 +0000 UTC m=+37.169109500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs") pod "router-default-578dddfc57-dnhg6" (UID: "18604274-740f-4f0b-bc1c-fb45e29fdc12") : secret "router-metrics-certs-default" not found Jul 03 14:09:18.313833 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.313756 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdde4f72-8103-49d7-b324-366692ef7c99-config\") pod \"console-operator-68b757fb67-6fj84\" (UID: \"cdde4f72-8103-49d7-b324-366692ef7c99\") " pod="openshift-console-operator/console-operator-68b757fb67-6fj84" Jul 03 14:09:18.313833 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.313882 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cdde4f72-8103-49d7-b324-366692ef7c99-trusted-ca\") pod \"console-operator-68b757fb67-6fj84\" (UID: \"cdde4f72-8103-49d7-b324-366692ef7c99\") " pod="openshift-console-operator/console-operator-68b757fb67-6fj84" Jul 03 14:09:18.315858 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.315833 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdde4f72-8103-49d7-b324-366692ef7c99-serving-cert\") pod \"console-operator-68b757fb67-6fj84\" (UID: \"cdde4f72-8103-49d7-b324-366692ef7c99\") " pod="openshift-console-operator/console-operator-68b757fb67-6fj84" Jul 03 14:09:18.315959 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.315865 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-stats-auth\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:18.316013 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.315973 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-default-certificate\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:18.323121 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.323077 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smwkl\" (UniqueName: \"kubernetes.io/projected/cdde4f72-8103-49d7-b324-366692ef7c99-kube-api-access-smwkl\") pod \"console-operator-68b757fb67-6fj84\" (UID: \"cdde4f72-8103-49d7-b324-366692ef7c99\") " pod="openshift-console-operator/console-operator-68b757fb67-6fj84" Jul 03 14:09:18.324621 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.324599 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ztn9\" (UniqueName: \"kubernetes.io/projected/18604274-740f-4f0b-bc1c-fb45e29fdc12-kube-api-access-8ztn9\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:18.324911 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.324887 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8bt5\" (UniqueName: \"kubernetes.io/projected/baef0a27-3024-4690-bb09-fee580be8129-kube-api-access-r8bt5\") pod \"cluster-samples-operator-56fcf86bf-j8nfw\" (UID: \"baef0a27-3024-4690-bb09-fee580be8129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw" Jul 03 14:09:18.386618 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.386369 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-6fj84" Jul 03 14:09:18.453109 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.453058 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tn4pv" event={"ID":"68e8c775-86ec-4944-9379-70d1bcfdbf69","Type":"ContainerStarted","Data":"5644f502dcf60936c19a8d4de0ca6d2d050d67f9654cd17eb34767812d0396a6"} Jul 03 14:09:18.585215 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.585131 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-6fj84"] Jul 03 14:09:18.589501 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:09:18.589467 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdde4f72_8103_49d7_b324_366692ef7c99.slice/crio-e86a875221f9ec5e15f92e270d8fc2ff43ef54524290f289ffa6d76904c65f61 WatchSource:0}: Error finding container e86a875221f9ec5e15f92e270d8fc2ff43ef54524290f289ffa6d76904c65f61: Status 404 returned error can't find the container with id e86a875221f9ec5e15f92e270d8fc2ff43ef54524290f289ffa6d76904c65f61 Jul 03 14:09:18.716044 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.716006 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls\") pod \"dns-default-wx4zj\" (UID: \"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4\") " pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:18.716209 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:18.716161 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 14:09:18.716278 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:18.716246 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls podName:2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:20.716219613 +0000 UTC m=+39.072504223 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls") pod "dns-default-wx4zj" (UID: "2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4") : secret "dns-default-metrics-tls" not found Jul 03 14:09:18.817507 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.817457 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:18.817716 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.817548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/baef0a27-3024-4690-bb09-fee580be8129-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-j8nfw\" (UID: \"baef0a27-3024-4690-bb09-fee580be8129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw" Jul 03 14:09:18.817716 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:18.817643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:18.817847 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:18.817729 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 03 14:09:18.817847 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:18.817803 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baef0a27-3024-4690-bb09-fee580be8129-samples-operator-tls podName:baef0a27-3024-4690-bb09-fee580be8129 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:19.81778242 +0000 UTC m=+38.174067035 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/baef0a27-3024-4690-bb09-fee580be8129-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-j8nfw" (UID: "baef0a27-3024-4690-bb09-fee580be8129") : secret "samples-operator-tls" not found Jul 03 14:09:18.817847 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:18.817807 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 03 14:09:18.817979 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:18.817862 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs podName:18604274-740f-4f0b-bc1c-fb45e29fdc12 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:19.817846529 +0000 UTC m=+38.174131129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs") pod "router-default-578dddfc57-dnhg6" (UID: "18604274-740f-4f0b-bc1c-fb45e29fdc12") : secret "router-metrics-certs-default" not found Jul 03 14:09:18.817979 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:18.817962 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle podName:18604274-740f-4f0b-bc1c-fb45e29fdc12 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:19.81794845 +0000 UTC m=+38.174233063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle") pod "router-default-578dddfc57-dnhg6" (UID: "18604274-740f-4f0b-bc1c-fb45e29fdc12") : configmap references non-existent config key: service-ca.crt Jul 03 14:09:19.456834 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:19.456781 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-6fj84" event={"ID":"cdde4f72-8103-49d7-b324-366692ef7c99","Type":"ContainerStarted","Data":"e86a875221f9ec5e15f92e270d8fc2ff43ef54524290f289ffa6d76904c65f61"} Jul 03 14:09:19.828251 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:19.827889 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:19.828251 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:19.827988 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/baef0a27-3024-4690-bb09-fee580be8129-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-j8nfw\" (UID: \"baef0a27-3024-4690-bb09-fee580be8129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw" Jul 03 14:09:19.828251 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:19.828032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:19.828251 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:19.828170 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle podName:18604274-740f-4f0b-bc1c-fb45e29fdc12 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:21.82814874 +0000 UTC m=+40.184433352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle") pod "router-default-578dddfc57-dnhg6" (UID: "18604274-740f-4f0b-bc1c-fb45e29fdc12") : configmap references non-existent config key: service-ca.crt Jul 03 14:09:19.828251 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:19.828179 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 03 14:09:19.828251 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:19.828208 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 03 14:09:19.828251 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:19.828224 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs podName:18604274-740f-4f0b-bc1c-fb45e29fdc12 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:21.828213092 +0000 UTC m=+40.184497704 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs") pod "router-default-578dddfc57-dnhg6" (UID: "18604274-740f-4f0b-bc1c-fb45e29fdc12") : secret "router-metrics-certs-default" not found Jul 03 14:09:19.828868 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:19.828282 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baef0a27-3024-4690-bb09-fee580be8129-samples-operator-tls podName:baef0a27-3024-4690-bb09-fee580be8129 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:21.82826787 +0000 UTC m=+40.184552468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/baef0a27-3024-4690-bb09-fee580be8129-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-j8nfw" (UID: "baef0a27-3024-4690-bb09-fee580be8129") : secret "samples-operator-tls" not found Jul 03 14:09:20.734968 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:20.734925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls\") pod \"dns-default-wx4zj\" (UID: \"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4\") " pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:20.735138 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:20.735109 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 14:09:20.735228 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:20.735205 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls podName:2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:24.735182251 +0000 UTC m=+43.091466852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls") pod "dns-default-wx4zj" (UID: "2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4") : secret "dns-default-metrics-tls" not found Jul 03 14:09:21.823774 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:21.823573 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v5kvh"] Jul 03 14:09:21.843753 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:21.843716 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:21.843894 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:21.843779 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/baef0a27-3024-4690-bb09-fee580be8129-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-j8nfw\" (UID: \"baef0a27-3024-4690-bb09-fee580be8129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw" Jul 03 14:09:21.843894 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:21.843824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:21.843975 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:21.843896 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle podName:18604274-740f-4f0b-bc1c-fb45e29fdc12 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:25.843876635 +0000 UTC m=+44.200161233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle") pod "router-default-578dddfc57-dnhg6" (UID: "18604274-740f-4f0b-bc1c-fb45e29fdc12") : configmap references non-existent config key: service-ca.crt Jul 03 14:09:21.843975 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:21.843953 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 03 14:09:21.843975 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:21.843955 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 03 14:09:21.844083 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:21.843999 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs podName:18604274-740f-4f0b-bc1c-fb45e29fdc12 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:25.843987344 +0000 UTC m=+44.200271959 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs") pod "router-default-578dddfc57-dnhg6" (UID: "18604274-740f-4f0b-bc1c-fb45e29fdc12") : secret "router-metrics-certs-default" not found Jul 03 14:09:21.844083 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:21.844016 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baef0a27-3024-4690-bb09-fee580be8129-samples-operator-tls podName:baef0a27-3024-4690-bb09-fee580be8129 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:25.844006209 +0000 UTC m=+44.200290813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/baef0a27-3024-4690-bb09-fee580be8129-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-j8nfw" (UID: "baef0a27-3024-4690-bb09-fee580be8129") : secret "samples-operator-tls" not found Jul 03 14:09:21.846980 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:21.846955 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:21.847961 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:21.847938 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v5kvh"] Jul 03 14:09:21.852705 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:21.852687 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 03 14:09:21.854319 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:21.854301 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kwmd4\"" Jul 03 14:09:21.855026 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:21.855008 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 03 14:09:21.855089 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:21.855010 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 03 14:09:21.855767 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:21.855750 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 03 14:09:21.945071 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:21.945043 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrmwv\" (UniqueName: \"kubernetes.io/projected/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-api-access-jrmwv\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:21.945071 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:21.945082 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc9da1fd-c331-406d-a767-5b50b53f696f-data-volume\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:21.945324 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:21.945104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc9da1fd-c331-406d-a767-5b50b53f696f-crio-socket\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:21.945324 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:21.945273 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:21.945414 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:21.945330 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:22.046156 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:22.046119 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrmwv\" (UniqueName: \"kubernetes.io/projected/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-api-access-jrmwv\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:22.046156 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:22.046161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc9da1fd-c331-406d-a767-5b50b53f696f-data-volume\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:22.046389 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:22.046181 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc9da1fd-c331-406d-a767-5b50b53f696f-crio-socket\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:22.046389 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:22.046342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:22.046500 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:22.046403 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc9da1fd-c331-406d-a767-5b50b53f696f-crio-socket\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:22.046500 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:22.046400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:22.046500 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:22.046461 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 03 14:09:22.046634 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:22.046498 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 14:09:22.046634 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:22.046510 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc9da1fd-c331-406d-a767-5b50b53f696f-data-volume\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:22.046634 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:22.046531 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-rbac-proxy-cm podName:cc9da1fd-c331-406d-a767-5b50b53f696f nodeName:}" failed. No retries permitted until 2026-07-03 14:09:22.546511249 +0000 UTC m=+40.902795865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v5kvh" (UID: "cc9da1fd-c331-406d-a767-5b50b53f696f") : configmap "kube-rbac-proxy" not found Jul 03 14:09:22.046634 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:22.046556 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls podName:cc9da1fd-c331-406d-a767-5b50b53f696f nodeName:}" failed. No retries permitted until 2026-07-03 14:09:22.546539402 +0000 UTC m=+40.902824019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v5kvh" (UID: "cc9da1fd-c331-406d-a767-5b50b53f696f") : secret "insights-runtime-extractor-tls" not found Jul 03 14:09:22.061606 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:22.061582 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrmwv\" (UniqueName: \"kubernetes.io/projected/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-api-access-jrmwv\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:22.477204 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:22.477168 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-ptfwd"] Jul 03 14:09:22.503190 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:22.503154 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-ptfwd"] Jul 03 14:09:22.503345 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:22.503282 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ptfwd" Jul 03 14:09:22.507946 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:22.507922 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qhtz9\"" Jul 03 14:09:22.549564 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:22.549534 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:22.549720 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:22.549582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:22.549720 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:22.549713 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 03 14:09:22.549799 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:22.549729 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 14:09:22.549799 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:22.549783 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls podName:cc9da1fd-c331-406d-a767-5b50b53f696f nodeName:}" failed. No retries permitted until 2026-07-03 14:09:23.549763494 +0000 UTC m=+41.906048093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v5kvh" (UID: "cc9da1fd-c331-406d-a767-5b50b53f696f") : secret "insights-runtime-extractor-tls" not found Jul 03 14:09:22.549799 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:22.549799 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-rbac-proxy-cm podName:cc9da1fd-c331-406d-a767-5b50b53f696f nodeName:}" failed. No retries permitted until 2026-07-03 14:09:23.549790631 +0000 UTC m=+41.906075230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v5kvh" (UID: "cc9da1fd-c331-406d-a767-5b50b53f696f") : configmap "kube-rbac-proxy" not found Jul 03 14:09:22.650782 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:22.650742 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjjjw\" (UniqueName: \"kubernetes.io/projected/a02fb905-12a8-44c4-9edf-7f483173fc8c-kube-api-access-mjjjw\") pod \"network-check-source-989f8b8dc-ptfwd\" (UID: \"a02fb905-12a8-44c4-9edf-7f483173fc8c\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ptfwd" Jul 03 14:09:22.751878 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:22.751791 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjjjw\" (UniqueName: \"kubernetes.io/projected/a02fb905-12a8-44c4-9edf-7f483173fc8c-kube-api-access-mjjjw\") pod \"network-check-source-989f8b8dc-ptfwd\" (UID: \"a02fb905-12a8-44c4-9edf-7f483173fc8c\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ptfwd" Jul 03 14:09:22.764287 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:22.764252 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjjjw\" (UniqueName: \"kubernetes.io/projected/a02fb905-12a8-44c4-9edf-7f483173fc8c-kube-api-access-mjjjw\") pod \"network-check-source-989f8b8dc-ptfwd\" (UID: \"a02fb905-12a8-44c4-9edf-7f483173fc8c\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ptfwd" Jul 03 14:09:22.815096 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:22.815065 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ptfwd" Jul 03 14:09:23.191362 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:23.191336 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-ptfwd"] Jul 03 14:09:23.207281 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:09:23.207239 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02fb905_12a8_44c4_9edf_7f483173fc8c.slice/crio-84139a4cf0ae9339a1140d3476b282c562e9234df865bcef078a9ebf34c9c8e7 WatchSource:0}: Error finding container 84139a4cf0ae9339a1140d3476b282c562e9234df865bcef078a9ebf34c9c8e7: Status 404 returned error can't find the container with id 84139a4cf0ae9339a1140d3476b282c562e9234df865bcef078a9ebf34c9c8e7 Jul 03 14:09:23.467728 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:23.467698 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/0.log" Jul 03 14:09:23.467886 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:23.467739 2576 generic.go:356] "Generic (PLEG): container finished" podID="cdde4f72-8103-49d7-b324-366692ef7c99" containerID="f15a9428b57dff23723c76a897504c44b1a6fe76987260bc1f4fd31884fba844" exitCode=255 Jul 03 14:09:23.467886 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:23.467819 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-6fj84" event={"ID":"cdde4f72-8103-49d7-b324-366692ef7c99","Type":"ContainerDied","Data":"f15a9428b57dff23723c76a897504c44b1a6fe76987260bc1f4fd31884fba844"} Jul 03 14:09:23.468038 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:23.468023 2576 scope.go:117] "RemoveContainer" containerID="f15a9428b57dff23723c76a897504c44b1a6fe76987260bc1f4fd31884fba844" Jul 03 14:09:23.469805 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:23.469780 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tn4pv" event={"ID":"68e8c775-86ec-4944-9379-70d1bcfdbf69","Type":"ContainerStarted","Data":"8eaa4ffd6ca1c2e11c8fbc5d33c9dc7e39aa364cb8f810826b64374d32b2bec8"} Jul 03 14:09:23.470963 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:23.470939 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ptfwd" event={"ID":"a02fb905-12a8-44c4-9edf-7f483173fc8c","Type":"ContainerStarted","Data":"84139a4cf0ae9339a1140d3476b282c562e9234df865bcef078a9ebf34c9c8e7"} Jul 03 14:09:23.473534 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:23.473512 2576 generic.go:356] "Generic (PLEG): container finished" podID="f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d" containerID="e500f7e58d5d191588a12138c6fc5f6db96583857a7695dbdc6651b3f1fcffa6" exitCode=0 Jul 03 14:09:23.473616 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:23.473546 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fvf88" event={"ID":"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d","Type":"ContainerDied","Data":"e500f7e58d5d191588a12138c6fc5f6db96583857a7695dbdc6651b3f1fcffa6"} Jul 03 14:09:23.553286 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:23.553211 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tn4pv" podStartSLOduration=2.021475461 podStartE2EDuration="7.553189585s" totalImagesPullingTime="5.531714124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:23.552934737 +0000 UTC m=+41.909219357" watchObservedRunningTime="2026-07-03 14:09:23.553189585 +0000 UTC m=+41.909474199" Jul 03 14:09:23.557996 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:23.557977 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:23.558087 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:23.558066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:23.558132 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:23.558113 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 03 14:09:23.558194 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:23.558183 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 14:09:23.558241 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:23.558184 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-rbac-proxy-cm podName:cc9da1fd-c331-406d-a767-5b50b53f696f nodeName:}" failed. No retries permitted until 2026-07-03 14:09:25.5581669 +0000 UTC m=+43.914451498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v5kvh" (UID: "cc9da1fd-c331-406d-a767-5b50b53f696f") : configmap "kube-rbac-proxy" not found Jul 03 14:09:23.558241 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:23.558232 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls podName:cc9da1fd-c331-406d-a767-5b50b53f696f nodeName:}" failed. No retries permitted until 2026-07-03 14:09:25.558220237 +0000 UTC m=+43.914504834 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v5kvh" (UID: "cc9da1fd-c331-406d-a767-5b50b53f696f") : secret "insights-runtime-extractor-tls" not found Jul 03 14:09:23.601921 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:23.601896 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4v7fc_928b4e88-1d06-41e9-99c9-efecc91c0ee9/dns-node-resolver/0.log" Jul 03 14:09:24.480363 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:24.480286 2576 generic.go:356] "Generic (PLEG): container finished" podID="f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d" containerID="7147ccb83c80a617ae6fa1b262f56068c9d6fa2eaba55f8850ab37129644e4c5" exitCode=0 Jul 03 14:09:24.480363 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:24.480345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fvf88" event={"ID":"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d","Type":"ContainerDied","Data":"7147ccb83c80a617ae6fa1b262f56068c9d6fa2eaba55f8850ab37129644e4c5"} Jul 03 14:09:24.482276 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:24.482255 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:09:24.482730 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:24.482706 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/0.log" Jul 03 14:09:24.482823 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:24.482745 2576 generic.go:356] "Generic (PLEG): container finished" podID="cdde4f72-8103-49d7-b324-366692ef7c99" containerID="e5e2a754aa6d01345596e7a0641e5fac74807724ea33f11bb63466e2ba0578ea" exitCode=255 Jul 03 14:09:24.482823 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:24.482780 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-6fj84" event={"ID":"cdde4f72-8103-49d7-b324-366692ef7c99","Type":"ContainerDied","Data":"e5e2a754aa6d01345596e7a0641e5fac74807724ea33f11bb63466e2ba0578ea"} Jul 03 14:09:24.482902 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:24.482857 2576 scope.go:117] "RemoveContainer" containerID="f15a9428b57dff23723c76a897504c44b1a6fe76987260bc1f4fd31884fba844" Jul 03 14:09:24.482995 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:24.482981 2576 scope.go:117] "RemoveContainer" containerID="e5e2a754aa6d01345596e7a0641e5fac74807724ea33f11bb63466e2ba0578ea" Jul 03 14:09:24.483143 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:24.483124 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-6fj84_openshift-console-operator(cdde4f72-8103-49d7-b324-366692ef7c99)\"" pod="openshift-console-operator/console-operator-68b757fb67-6fj84" podUID="cdde4f72-8103-49d7-b324-366692ef7c99" Jul 03 14:09:24.769636 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:24.769540 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls\") pod \"dns-default-wx4zj\" (UID: \"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4\") " pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:24.769806 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:24.769736 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 14:09:24.769853 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:24.769815 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls podName:2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:32.76979466 +0000 UTC m=+51.126079261 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls") pod "dns-default-wx4zj" (UID: "2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4") : secret "dns-default-metrics-tls" not found Jul 03 14:09:24.803101 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:24.803070 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hjwhk_b6e59bf8-184d-4593-aa5f-de1c5854ac77/node-ca/0.log" Jul 03 14:09:25.489706 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:25.489671 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fvf88" event={"ID":"f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d","Type":"ContainerStarted","Data":"4badf6d142a1694e1ccab7db23ff911665f014f8efe80aa4911fea9a6c76b8bb"} Jul 03 14:09:25.491334 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:25.491311 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:09:25.491670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:25.491636 2576 scope.go:117] "RemoveContainer" containerID="e5e2a754aa6d01345596e7a0641e5fac74807724ea33f11bb63466e2ba0578ea" Jul 03 14:09:25.491839 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:25.491819 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-6fj84_openshift-console-operator(cdde4f72-8103-49d7-b324-366692ef7c99)\"" pod="openshift-console-operator/console-operator-68b757fb67-6fj84" podUID="cdde4f72-8103-49d7-b324-366692ef7c99" Jul 03 14:09:25.523847 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:25.523796 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fvf88" podStartSLOduration=10.215530477 podStartE2EDuration="43.523776882s" totalImagesPullingTime="33.308246405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 14:08:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:25.521971897 +0000 UTC m=+43.878256519" watchObservedRunningTime="2026-07-03 14:09:25.523776882 +0000 UTC m=+43.880061505" Jul 03 14:09:25.577105 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:25.577069 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:25.577262 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:25.577130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:25.577262 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:25.577214 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 03 14:09:25.577388 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:25.577296 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-rbac-proxy-cm podName:cc9da1fd-c331-406d-a767-5b50b53f696f nodeName:}" failed. No retries permitted until 2026-07-03 14:09:29.577276211 +0000 UTC m=+47.933560811 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v5kvh" (UID: "cc9da1fd-c331-406d-a767-5b50b53f696f") : configmap "kube-rbac-proxy" not found Jul 03 14:09:25.577388 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:25.577313 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 14:09:25.577388 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:25.577387 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls podName:cc9da1fd-c331-406d-a767-5b50b53f696f nodeName:}" failed. No retries permitted until 2026-07-03 14:09:29.577370296 +0000 UTC m=+47.933654901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v5kvh" (UID: "cc9da1fd-c331-406d-a767-5b50b53f696f") : secret "insights-runtime-extractor-tls" not found Jul 03 14:09:25.802973 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:25.802925 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tn4pv_68e8c775-86ec-4944-9379-70d1bcfdbf69/serve-healthcheck-canary/0.log" Jul 03 14:09:25.880638 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:25.880616 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:25.880722 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:25.880711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/baef0a27-3024-4690-bb09-fee580be8129-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-j8nfw\" (UID: \"baef0a27-3024-4690-bb09-fee580be8129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw" Jul 03 14:09:25.880771 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:25.880750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:25.880879 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:25.880801 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle podName:18604274-740f-4f0b-bc1c-fb45e29fdc12 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:33.880776076 +0000 UTC m=+52.237060686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle") pod "router-default-578dddfc57-dnhg6" (UID: "18604274-740f-4f0b-bc1c-fb45e29fdc12") : configmap references non-existent config key: service-ca.crt Jul 03 14:09:25.880879 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:25.880852 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 03 14:09:25.880879 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:25.880852 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 03 14:09:25.881007 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:25.880906 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baef0a27-3024-4690-bb09-fee580be8129-samples-operator-tls podName:baef0a27-3024-4690-bb09-fee580be8129 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:33.880889976 +0000 UTC m=+52.237174579 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/baef0a27-3024-4690-bb09-fee580be8129-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-j8nfw" (UID: "baef0a27-3024-4690-bb09-fee580be8129") : secret "samples-operator-tls" not found Jul 03 14:09:25.881007 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:25.880919 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs podName:18604274-740f-4f0b-bc1c-fb45e29fdc12 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:33.88091277 +0000 UTC m=+52.237197368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs") pod "router-default-578dddfc57-dnhg6" (UID: "18604274-740f-4f0b-bc1c-fb45e29fdc12") : secret "router-metrics-certs-default" not found Jul 03 14:09:26.495404 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:26.495366 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ptfwd" event={"ID":"a02fb905-12a8-44c4-9edf-7f483173fc8c","Type":"ContainerStarted","Data":"48c0bdcf6785e2a60d04585866ceeedcc793fdaa695ed9dc0b9f15eb9e2d960d"} Jul 03 14:09:28.387373 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.387334 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-6fj84" Jul 03 14:09:28.387772 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.387392 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-6fj84" Jul 03 14:09:28.387772 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.387746 2576 scope.go:117] "RemoveContainer" containerID="e5e2a754aa6d01345596e7a0641e5fac74807724ea33f11bb63466e2ba0578ea" Jul 03 14:09:28.387906 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:28.387887 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-6fj84_openshift-console-operator(cdde4f72-8103-49d7-b324-366692ef7c99)\"" pod="openshift-console-operator/console-operator-68b757fb67-6fj84" podUID="cdde4f72-8103-49d7-b324-366692ef7c99" Jul 03 14:09:28.655827 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.655733 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ptfwd" podStartSLOduration=3.986611823 podStartE2EDuration="6.65571002s" totalImagesPullingTime="2.669098197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:26.516533425 +0000 UTC m=+44.872818045" watchObservedRunningTime="2026-07-03 14:09:28.65571002 +0000 UTC m=+47.011994691" Jul 03 14:09:28.655973 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.655852 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-nzgfw"] Jul 03 14:09:28.658623 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.658608 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-nzgfw" Jul 03 14:09:28.664192 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.664156 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 03 14:09:28.664627 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.664596 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 03 14:09:28.665608 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.665588 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 03 14:09:28.665608 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.665598 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-grz4n\"" Jul 03 14:09:28.665787 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.665593 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 03 14:09:28.670224 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.670202 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-nzgfw"] Jul 03 14:09:28.703666 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.703636 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c42d12a0-d089-4bc9-9310-5c0de1bab5a0-signing-key\") pod \"service-ca-758f599c5-nzgfw\" (UID: \"c42d12a0-d089-4bc9-9310-5c0de1bab5a0\") " pod="openshift-service-ca/service-ca-758f599c5-nzgfw" Jul 03 14:09:28.703802 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.703691 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c42d12a0-d089-4bc9-9310-5c0de1bab5a0-signing-cabundle\") pod \"service-ca-758f599c5-nzgfw\" (UID: \"c42d12a0-d089-4bc9-9310-5c0de1bab5a0\") " pod="openshift-service-ca/service-ca-758f599c5-nzgfw" Jul 03 14:09:28.703802 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.703721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47p28\" (UniqueName: \"kubernetes.io/projected/c42d12a0-d089-4bc9-9310-5c0de1bab5a0-kube-api-access-47p28\") pod \"service-ca-758f599c5-nzgfw\" (UID: \"c42d12a0-d089-4bc9-9310-5c0de1bab5a0\") " pod="openshift-service-ca/service-ca-758f599c5-nzgfw" Jul 03 14:09:28.804932 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.804885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c42d12a0-d089-4bc9-9310-5c0de1bab5a0-signing-key\") pod \"service-ca-758f599c5-nzgfw\" (UID: \"c42d12a0-d089-4bc9-9310-5c0de1bab5a0\") " pod="openshift-service-ca/service-ca-758f599c5-nzgfw" Jul 03 14:09:28.805086 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.804947 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c42d12a0-d089-4bc9-9310-5c0de1bab5a0-signing-cabundle\") pod \"service-ca-758f599c5-nzgfw\" (UID: \"c42d12a0-d089-4bc9-9310-5c0de1bab5a0\") " pod="openshift-service-ca/service-ca-758f599c5-nzgfw" Jul 03 14:09:28.805086 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.804995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47p28\" (UniqueName: \"kubernetes.io/projected/c42d12a0-d089-4bc9-9310-5c0de1bab5a0-kube-api-access-47p28\") pod \"service-ca-758f599c5-nzgfw\" (UID: \"c42d12a0-d089-4bc9-9310-5c0de1bab5a0\") " pod="openshift-service-ca/service-ca-758f599c5-nzgfw" Jul 03 14:09:28.805672 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.805633 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c42d12a0-d089-4bc9-9310-5c0de1bab5a0-signing-cabundle\") pod \"service-ca-758f599c5-nzgfw\" (UID: \"c42d12a0-d089-4bc9-9310-5c0de1bab5a0\") " pod="openshift-service-ca/service-ca-758f599c5-nzgfw" Jul 03 14:09:28.807488 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.807455 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c42d12a0-d089-4bc9-9310-5c0de1bab5a0-signing-key\") pod \"service-ca-758f599c5-nzgfw\" (UID: \"c42d12a0-d089-4bc9-9310-5c0de1bab5a0\") " pod="openshift-service-ca/service-ca-758f599c5-nzgfw" Jul 03 14:09:28.819208 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.819183 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47p28\" (UniqueName: \"kubernetes.io/projected/c42d12a0-d089-4bc9-9310-5c0de1bab5a0-kube-api-access-47p28\") pod \"service-ca-758f599c5-nzgfw\" (UID: \"c42d12a0-d089-4bc9-9310-5c0de1bab5a0\") " pod="openshift-service-ca/service-ca-758f599c5-nzgfw" Jul 03 14:09:28.967571 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:28.967502 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-nzgfw" Jul 03 14:09:29.097969 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:29.097941 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-nzgfw"] Jul 03 14:09:29.100625 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:09:29.100598 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42d12a0_d089_4bc9_9310_5c0de1bab5a0.slice/crio-878b9f53f79a283d173b44593403f035465e06463c6e31d6dc19845e684afc12 WatchSource:0}: Error finding container 878b9f53f79a283d173b44593403f035465e06463c6e31d6dc19845e684afc12: Status 404 returned error can't find the container with id 878b9f53f79a283d173b44593403f035465e06463c6e31d6dc19845e684afc12 Jul 03 14:09:29.503900 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:29.503863 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-nzgfw" event={"ID":"c42d12a0-d089-4bc9-9310-5c0de1bab5a0","Type":"ContainerStarted","Data":"878b9f53f79a283d173b44593403f035465e06463c6e31d6dc19845e684afc12"} Jul 03 14:09:29.611911 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:29.611872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:29.612111 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:29.611926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:29.612111 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:29.612011 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 03 14:09:29.612111 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:29.612092 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-rbac-proxy-cm podName:cc9da1fd-c331-406d-a767-5b50b53f696f nodeName:}" failed. No retries permitted until 2026-07-03 14:09:37.612072211 +0000 UTC m=+55.968356824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v5kvh" (UID: "cc9da1fd-c331-406d-a767-5b50b53f696f") : configmap "kube-rbac-proxy" not found Jul 03 14:09:29.612111 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:29.612094 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 14:09:29.612320 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:29.612149 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls podName:cc9da1fd-c331-406d-a767-5b50b53f696f nodeName:}" failed. No retries permitted until 2026-07-03 14:09:37.612131109 +0000 UTC m=+55.968415713 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v5kvh" (UID: "cc9da1fd-c331-406d-a767-5b50b53f696f") : secret "insights-runtime-extractor-tls" not found Jul 03 14:09:32.512226 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:32.512184 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-nzgfw" event={"ID":"c42d12a0-d089-4bc9-9310-5c0de1bab5a0","Type":"ContainerStarted","Data":"77f06910bfd1860d439312d39f366114de984898cdc42e40a38fee6b66929e88"} Jul 03 14:09:32.535720 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:32.535671 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-nzgfw" podStartSLOduration=2.043391627 podStartE2EDuration="4.535642275s" totalImagesPullingTime="2.492250648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:32.53512358 +0000 UTC m=+50.891408200" watchObservedRunningTime="2026-07-03 14:09:32.535642275 +0000 UTC m=+50.891926912" Jul 03 14:09:32.842970 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:32.842931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls\") pod \"dns-default-wx4zj\" (UID: \"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4\") " pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:32.843188 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:32.843091 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 14:09:32.843188 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:32.843156 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls podName:2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:48.843138251 +0000 UTC m=+67.199422854 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls") pod "dns-default-wx4zj" (UID: "2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4") : secret "dns-default-metrics-tls" not found Jul 03 14:09:33.951301 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:33.951264 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:33.951771 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:33.951340 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:33.951771 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:33.951392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/baef0a27-3024-4690-bb09-fee580be8129-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-j8nfw\" (UID: \"baef0a27-3024-4690-bb09-fee580be8129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw" Jul 03 14:09:33.951771 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:33.951453 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 03 14:09:33.951771 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:33.951465 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle podName:18604274-740f-4f0b-bc1c-fb45e29fdc12 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:49.951447506 +0000 UTC m=+68.307732106 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle") pod "router-default-578dddfc57-dnhg6" (UID: "18604274-740f-4f0b-bc1c-fb45e29fdc12") : configmap references non-existent config key: service-ca.crt Jul 03 14:09:33.951771 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:33.951525 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs podName:18604274-740f-4f0b-bc1c-fb45e29fdc12 nodeName:}" failed. No retries permitted until 2026-07-03 14:09:49.951508126 +0000 UTC m=+68.307792728 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs") pod "router-default-578dddfc57-dnhg6" (UID: "18604274-740f-4f0b-bc1c-fb45e29fdc12") : secret "router-metrics-certs-default" not found Jul 03 14:09:33.953826 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:33.953808 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/baef0a27-3024-4690-bb09-fee580be8129-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-j8nfw\" (UID: \"baef0a27-3024-4690-bb09-fee580be8129\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw" Jul 03 14:09:33.970789 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:33.970757 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw" Jul 03 14:09:34.166981 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:34.166961 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw"] Jul 03 14:09:34.519314 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:34.519226 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw" event={"ID":"baef0a27-3024-4690-bb09-fee580be8129","Type":"ContainerStarted","Data":"1003589b627a02a8fe58aa41f6a8e6a81ab05a5c3c8083d3a13a810b139586b4"} Jul 03 14:09:37.530011 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:37.529971 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw" event={"ID":"baef0a27-3024-4690-bb09-fee580be8129","Type":"ContainerStarted","Data":"56e33c23321276e8bedf436ff7ede86a005a7a21d6766b7a850b760ed30038ea"} Jul 03 14:09:37.530501 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:37.530033 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw" event={"ID":"baef0a27-3024-4690-bb09-fee580be8129","Type":"ContainerStarted","Data":"65a4a4002f08de6c7b7238134549728efb5c4870498d9bda1e7f3b06f681b684"} Jul 03 14:09:37.554912 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:37.554858 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-j8nfw" podStartSLOduration=17.158299406 podStartE2EDuration="19.554838707s" totalImagesPullingTime="2.396539301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:37.554685758 +0000 UTC m=+55.910970383" watchObservedRunningTime="2026-07-03 14:09:37.554838707 +0000 UTC m=+55.911123328" Jul 03 14:09:37.683032 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:37.682995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:37.683032 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:37.683038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:37.683258 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:37.683170 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 14:09:37.683258 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:09:37.683226 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls podName:cc9da1fd-c331-406d-a767-5b50b53f696f nodeName:}" failed. No retries permitted until 2026-07-03 14:09:53.683213038 +0000 UTC m=+72.039497636 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v5kvh" (UID: "cc9da1fd-c331-406d-a767-5b50b53f696f") : secret "insights-runtime-extractor-tls" not found Jul 03 14:09:37.686562 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:37.683995 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc9da1fd-c331-406d-a767-5b50b53f696f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:43.162081 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:43.162049 2576 scope.go:117] "RemoveContainer" containerID="e5e2a754aa6d01345596e7a0641e5fac74807724ea33f11bb63466e2ba0578ea" Jul 03 14:09:43.445714 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:43.445622 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtmd7" Jul 03 14:09:43.551126 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:43.551093 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:09:43.551278 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:43.551165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-6fj84" event={"ID":"cdde4f72-8103-49d7-b324-366692ef7c99","Type":"ContainerStarted","Data":"08c8502e5492a87275169937e61094bae0dfcf7de67777f2e6a0c302f8967380"} Jul 03 14:09:43.551465 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:43.551440 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-6fj84" Jul 03 14:09:43.587807 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:43.587759 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-6fj84" podStartSLOduration=21.13678105 podStartE2EDuration="25.587744714s" totalImagesPullingTime="4.450963664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:43.586208114 +0000 UTC m=+61.942492732" watchObservedRunningTime="2026-07-03 14:09:43.587744714 +0000 UTC m=+61.944029365" Jul 03 14:09:43.688067 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:43.688040 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-6fj84" Jul 03 14:09:47.872299 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:47.872255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnsvt\" (UniqueName: \"kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt\") pod \"network-check-target-6n5dr\" (UID: \"e48e7e11-89d3-4f42-be11-8f2b3c685504\") " pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:09:47.875101 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:47.875075 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnsvt\" (UniqueName: \"kubernetes.io/projected/e48e7e11-89d3-4f42-be11-8f2b3c685504-kube-api-access-cnsvt\") pod \"network-check-target-6n5dr\" (UID: \"e48e7e11-89d3-4f42-be11-8f2b3c685504\") " pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:09:47.973695 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:47.973631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs\") pod \"network-metrics-daemon-7n9kl\" (UID: \"514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb\") " pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:09:47.973862 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:47.973752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret\") pod \"global-pull-secret-syncer-xn7tc\" (UID: \"21624337-43ef-4f45-8118-63b9de2f9ed9\") " pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:09:47.977014 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:47.976991 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 03 14:09:47.978154 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:47.978138 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 03 14:09:47.986950 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:47.986921 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21624337-43ef-4f45-8118-63b9de2f9ed9-original-pull-secret\") pod \"global-pull-secret-syncer-xn7tc\" (UID: \"21624337-43ef-4f45-8118-63b9de2f9ed9\") " pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:09:47.987054 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:47.987037 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb-metrics-certs\") pod \"network-metrics-daemon-7n9kl\" (UID: \"514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb\") " pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:09:48.078080 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:48.078052 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tlbvf\"" Jul 03 14:09:48.085199 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:48.085177 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:09:48.087552 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:48.087265 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-472cc\"" Jul 03 14:09:48.089424 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:48.089406 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xn7tc" Jul 03 14:09:48.092159 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:48.092141 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n9kl" Jul 03 14:09:48.257495 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:48.257465 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6n5dr"] Jul 03 14:09:48.260425 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:09:48.260403 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48e7e11_89d3_4f42_be11_8f2b3c685504.slice/crio-1c9238eb535a599f54c00aef64a4d6b91f9f46bf964cfd05f966e0ef45eb90dd WatchSource:0}: Error finding container 1c9238eb535a599f54c00aef64a4d6b91f9f46bf964cfd05f966e0ef45eb90dd: Status 404 returned error can't find the container with id 1c9238eb535a599f54c00aef64a4d6b91f9f46bf964cfd05f966e0ef45eb90dd Jul 03 14:09:48.459961 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:48.459770 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xn7tc"] Jul 03 14:09:48.462345 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:48.462320 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7n9kl"] Jul 03 14:09:48.462537 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:09:48.462516 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21624337_43ef_4f45_8118_63b9de2f9ed9.slice/crio-3ce7467a9880386c42ea08469d9923d71b1c6c9b27639462a5dec19af767093b WatchSource:0}: Error finding container 3ce7467a9880386c42ea08469d9923d71b1c6c9b27639462a5dec19af767093b: Status 404 returned error can't find the container with id 3ce7467a9880386c42ea08469d9923d71b1c6c9b27639462a5dec19af767093b Jul 03 14:09:48.465085 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:09:48.465064 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514ffd5f_8ac2_4e0e_a22d_b4c843c1adbb.slice/crio-a8f8bf5930e85006f478358bec60e5590b8801b3bede519cafc90462f8e8ffd8 WatchSource:0}: Error finding container a8f8bf5930e85006f478358bec60e5590b8801b3bede519cafc90462f8e8ffd8: Status 404 returned error can't find the container with id a8f8bf5930e85006f478358bec60e5590b8801b3bede519cafc90462f8e8ffd8 Jul 03 14:09:48.567253 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:48.567209 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7n9kl" event={"ID":"514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb","Type":"ContainerStarted","Data":"a8f8bf5930e85006f478358bec60e5590b8801b3bede519cafc90462f8e8ffd8"} Jul 03 14:09:48.568284 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:48.568259 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xn7tc" event={"ID":"21624337-43ef-4f45-8118-63b9de2f9ed9","Type":"ContainerStarted","Data":"3ce7467a9880386c42ea08469d9923d71b1c6c9b27639462a5dec19af767093b"} Jul 03 14:09:48.569598 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:48.569576 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6n5dr" event={"ID":"e48e7e11-89d3-4f42-be11-8f2b3c685504","Type":"ContainerStarted","Data":"6e9ce71a5dfdd2e7de973b2c48bba9cd00903f75ac69adc6bd454642efc15540"} Jul 03 14:09:48.569716 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:48.569608 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6n5dr" event={"ID":"e48e7e11-89d3-4f42-be11-8f2b3c685504","Type":"ContainerStarted","Data":"1c9238eb535a599f54c00aef64a4d6b91f9f46bf964cfd05f966e0ef45eb90dd"} Jul 03 14:09:48.569716 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:48.569703 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:09:48.591995 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:48.591943 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6n5dr" podStartSLOduration=66.591926754 podStartE2EDuration="1m6.591926754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:48.590223991 +0000 UTC m=+66.946508610" watchObservedRunningTime="2026-07-03 14:09:48.591926754 +0000 UTC m=+66.948211383" Jul 03 14:09:48.882582 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:48.882506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls\") pod \"dns-default-wx4zj\" (UID: \"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4\") " pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:48.884987 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:48.884965 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4-metrics-tls\") pod \"dns-default-wx4zj\" (UID: \"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4\") " pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:49.082950 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:49.082917 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wf6pg\"" Jul 03 14:09:49.090522 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:49.090479 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:49.252308 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:49.252272 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wx4zj"] Jul 03 14:09:49.256914 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:09:49.256882 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0dda2d_eb99_478f_aca7_6f3b7f1a1dd4.slice/crio-bbdafceb24b857f2b42aa06509792888de567c4ccc0f4bcf5f23394ce5e9c978 WatchSource:0}: Error finding container bbdafceb24b857f2b42aa06509792888de567c4ccc0f4bcf5f23394ce5e9c978: Status 404 returned error can't find the container with id bbdafceb24b857f2b42aa06509792888de567c4ccc0f4bcf5f23394ce5e9c978 Jul 03 14:09:49.574593 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:49.574522 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wx4zj" event={"ID":"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4","Type":"ContainerStarted","Data":"bbdafceb24b857f2b42aa06509792888de567c4ccc0f4bcf5f23394ce5e9c978"} Jul 03 14:09:49.990545 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:49.990451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:49.990545 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:49.990535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:49.991305 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:49.991280 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18604274-740f-4f0b-bc1c-fb45e29fdc12-service-ca-bundle\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:49.994403 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:49.994378 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18604274-740f-4f0b-bc1c-fb45e29fdc12-metrics-certs\") pod \"router-default-578dddfc57-dnhg6\" (UID: \"18604274-740f-4f0b-bc1c-fb45e29fdc12\") " pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:50.213910 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:50.213881 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cvjmf\"" Jul 03 14:09:50.221009 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:50.220979 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:50.386346 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:50.386304 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-578dddfc57-dnhg6"] Jul 03 14:09:50.391671 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:09:50.390838 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18604274_740f_4f0b_bc1c_fb45e29fdc12.slice/crio-66629bbb0fa3161d92362dcad90228e2b52534b1c986d0608d4dbc1e2a432b7f WatchSource:0}: Error finding container 66629bbb0fa3161d92362dcad90228e2b52534b1c986d0608d4dbc1e2a432b7f: Status 404 returned error can't find the container with id 66629bbb0fa3161d92362dcad90228e2b52534b1c986d0608d4dbc1e2a432b7f Jul 03 14:09:50.581207 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:50.581016 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7n9kl" event={"ID":"514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb","Type":"ContainerStarted","Data":"2496ffd8f2791b234db15a1bc8f231bdf45f54ebe546ff9326cb0808251408fb"} Jul 03 14:09:50.581207 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:50.581089 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7n9kl" event={"ID":"514ffd5f-8ac2-4e0e-a22d-b4c843c1adbb","Type":"ContainerStarted","Data":"56571c53f12d0bfd71ed87dbb740ebb5bda51bd4212864002adaac11847beeb3"} Jul 03 14:09:50.582861 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:50.582811 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-578dddfc57-dnhg6" event={"ID":"18604274-740f-4f0b-bc1c-fb45e29fdc12","Type":"ContainerStarted","Data":"be8d0f5df6628b2e1afc357ae594566264924207bf194771de70f3f63760807b"} Jul 03 14:09:50.583020 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:50.582871 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-578dddfc57-dnhg6" event={"ID":"18604274-740f-4f0b-bc1c-fb45e29fdc12","Type":"ContainerStarted","Data":"66629bbb0fa3161d92362dcad90228e2b52534b1c986d0608d4dbc1e2a432b7f"} Jul 03 14:09:50.636124 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:50.636070 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-578dddfc57-dnhg6" podStartSLOduration=32.636054846 podStartE2EDuration="32.636054846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:50.635905261 +0000 UTC m=+68.992189881" watchObservedRunningTime="2026-07-03 14:09:50.636054846 +0000 UTC m=+68.992339465" Jul 03 14:09:50.637462 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:50.636921 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7n9kl" podStartSLOduration=67.529715589 podStartE2EDuration="1m8.636905042s" totalImagesPullingTime="1.107189453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:50.606163503 +0000 UTC m=+68.962448123" watchObservedRunningTime="2026-07-03 14:09:50.636905042 +0000 UTC m=+68.993189663" Jul 03 14:09:51.221398 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.221361 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:51.223933 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.223908 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:51.588997 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.588945 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wx4zj" event={"ID":"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4","Type":"ContainerStarted","Data":"9cb00541afd5cd858e6f45cf4a4e05934a553d2b17537015883befd8adaa2feb"} Jul 03 14:09:51.589179 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.589024 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wx4zj" event={"ID":"2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4","Type":"ContainerStarted","Data":"66a93ef5cfd16241f1a72ba0ad15ea028e4c814c74b035000f7e120135d721fb"} Jul 03 14:09:51.589471 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.589456 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:51.590725 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.590705 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-578dddfc57-dnhg6" Jul 03 14:09:51.657665 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.657631 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9777d88-5q287"] Jul 03 14:09:51.660532 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.660510 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9777d88-5q287" Jul 03 14:09:51.681554 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.681521 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 03 14:09:51.681894 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.681864 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 03 14:09:51.682039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.682021 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 03 14:09:51.682155 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.682135 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 03 14:09:51.682241 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.682224 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-d4n8x\"" Jul 03 14:09:51.699793 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.699727 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wx4zj" podStartSLOduration=33.961360777 podStartE2EDuration="35.699705482s" totalImagesPullingTime="1.738344705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:51.681868901 +0000 UTC m=+70.038153521" watchObservedRunningTime="2026-07-03 14:09:51.699705482 +0000 UTC m=+70.055990102" Jul 03 14:09:51.699963 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.699936 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9777d88-5q287"] Jul 03 14:09:51.806528 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.806485 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/83951aa5-0627-4ece-829a-91da7f3545e5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69d9777d88-5q287\" (UID: \"83951aa5-0627-4ece-829a-91da7f3545e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9777d88-5q287" Jul 03 14:09:51.806743 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.806542 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zknn\" (UniqueName: \"kubernetes.io/projected/83951aa5-0627-4ece-829a-91da7f3545e5-kube-api-access-2zknn\") pod \"managed-serviceaccount-addon-agent-69d9777d88-5q287\" (UID: \"83951aa5-0627-4ece-829a-91da7f3545e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9777d88-5q287" Jul 03 14:09:51.854293 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.854207 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d699c5c4-fv99h"] Jul 03 14:09:51.856965 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.856940 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:51.869053 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.869028 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 03 14:09:51.869376 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.869359 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 03 14:09:51.869543 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.869530 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 03 14:09:51.869745 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.869733 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-scvmw\"" Jul 03 14:09:51.877701 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.877679 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 03 14:09:51.901536 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.901502 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d699c5c4-fv99h"] Jul 03 14:09:51.907276 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.907229 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d1e565a7-e9d7-4b4c-bc53-147e9014748a-registry-certificates\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:51.907276 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.907266 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1e565a7-e9d7-4b4c-bc53-147e9014748a-registry-tls\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:51.908194 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.907305 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d1e565a7-e9d7-4b4c-bc53-147e9014748a-ca-trust-extracted\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:51.908194 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.907337 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/83951aa5-0627-4ece-829a-91da7f3545e5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69d9777d88-5q287\" (UID: \"83951aa5-0627-4ece-829a-91da7f3545e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9777d88-5q287" Jul 03 14:09:51.908194 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.907366 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zknn\" (UniqueName: \"kubernetes.io/projected/83951aa5-0627-4ece-829a-91da7f3545e5-kube-api-access-2zknn\") pod \"managed-serviceaccount-addon-agent-69d9777d88-5q287\" (UID: \"83951aa5-0627-4ece-829a-91da7f3545e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9777d88-5q287" Jul 03 14:09:51.908194 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.907393 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zdzq\" (UniqueName: \"kubernetes.io/projected/d1e565a7-e9d7-4b4c-bc53-147e9014748a-kube-api-access-6zdzq\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:51.908194 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.907419 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1e565a7-e9d7-4b4c-bc53-147e9014748a-bound-sa-token\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:51.908194 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.907458 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1e565a7-e9d7-4b4c-bc53-147e9014748a-trusted-ca\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:51.908194 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.907553 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d1e565a7-e9d7-4b4c-bc53-147e9014748a-installation-pull-secrets\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:51.908194 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.907640 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d1e565a7-e9d7-4b4c-bc53-147e9014748a-image-registry-private-configuration\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:51.913313 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.913287 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/83951aa5-0627-4ece-829a-91da7f3545e5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69d9777d88-5q287\" (UID: \"83951aa5-0627-4ece-829a-91da7f3545e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9777d88-5q287" Jul 03 14:09:51.946918 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.946886 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zknn\" (UniqueName: \"kubernetes.io/projected/83951aa5-0627-4ece-829a-91da7f3545e5-kube-api-access-2zknn\") pod \"managed-serviceaccount-addon-agent-69d9777d88-5q287\" (UID: \"83951aa5-0627-4ece-829a-91da7f3545e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9777d88-5q287" Jul 03 14:09:51.980201 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:51.980163 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9777d88-5q287" Jul 03 14:09:52.008933 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.008886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d1e565a7-e9d7-4b4c-bc53-147e9014748a-image-registry-private-configuration\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:52.009115 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.008947 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d1e565a7-e9d7-4b4c-bc53-147e9014748a-registry-certificates\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:52.009115 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.008977 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1e565a7-e9d7-4b4c-bc53-147e9014748a-registry-tls\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:52.009115 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.009006 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d1e565a7-e9d7-4b4c-bc53-147e9014748a-ca-trust-extracted\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:52.009115 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.009033 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zdzq\" (UniqueName: \"kubernetes.io/projected/d1e565a7-e9d7-4b4c-bc53-147e9014748a-kube-api-access-6zdzq\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:52.009115 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.009055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1e565a7-e9d7-4b4c-bc53-147e9014748a-bound-sa-token\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:52.009459 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.009114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1e565a7-e9d7-4b4c-bc53-147e9014748a-trusted-ca\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:52.009459 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.009184 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d1e565a7-e9d7-4b4c-bc53-147e9014748a-installation-pull-secrets\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:52.009581 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.009458 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d1e565a7-e9d7-4b4c-bc53-147e9014748a-ca-trust-extracted\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:52.009936 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.009915 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d1e565a7-e9d7-4b4c-bc53-147e9014748a-registry-certificates\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:52.010321 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.010297 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1e565a7-e9d7-4b4c-bc53-147e9014748a-trusted-ca\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:52.012597 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.012578 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d1e565a7-e9d7-4b4c-bc53-147e9014748a-image-registry-private-configuration\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:52.012719 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.012703 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1e565a7-e9d7-4b4c-bc53-147e9014748a-registry-tls\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:52.012776 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.012733 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d1e565a7-e9d7-4b4c-bc53-147e9014748a-installation-pull-secrets\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:52.024358 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.024333 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1e565a7-e9d7-4b4c-bc53-147e9014748a-bound-sa-token\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:52.026474 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.026452 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zdzq\" (UniqueName: \"kubernetes.io/projected/d1e565a7-e9d7-4b4c-bc53-147e9014748a-kube-api-access-6zdzq\") pod \"image-registry-5d699c5c4-fv99h\" (UID: \"d1e565a7-e9d7-4b4c-bc53-147e9014748a\") " pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:52.169006 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.168925 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:52.593127 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.593094 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wx4zj" Jul 03 14:09:52.902916 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.902845 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d699c5c4-fv99h"] Jul 03 14:09:52.906080 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:09:52.906052 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e565a7_e9d7_4b4c_bc53_147e9014748a.slice/crio-b46fb3124db1dc141dbaf988d9fbfef68abef0f943c5239a4023030d06c33913 WatchSource:0}: Error finding container b46fb3124db1dc141dbaf988d9fbfef68abef0f943c5239a4023030d06c33913: Status 404 returned error can't find the container with id b46fb3124db1dc141dbaf988d9fbfef68abef0f943c5239a4023030d06c33913 Jul 03 14:09:52.929799 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:52.929766 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9777d88-5q287"] Jul 03 14:09:52.931644 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:09:52.931622 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83951aa5_0627_4ece_829a_91da7f3545e5.slice/crio-43b6f0be6484bc7b404c4e102fab5ee94316638b87906bd003ceb6179d28e0b8 WatchSource:0}: Error finding container 43b6f0be6484bc7b404c4e102fab5ee94316638b87906bd003ceb6179d28e0b8: Status 404 returned error can't find the container with id 43b6f0be6484bc7b404c4e102fab5ee94316638b87906bd003ceb6179d28e0b8 Jul 03 14:09:53.597672 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:53.597626 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" event={"ID":"d1e565a7-e9d7-4b4c-bc53-147e9014748a","Type":"ContainerStarted","Data":"2623386c0c9a75e15c45e3ec84ba2a445694800ac9c85abf197e05803ddeab96"} Jul 03 14:09:53.598141 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:53.597701 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" event={"ID":"d1e565a7-e9d7-4b4c-bc53-147e9014748a","Type":"ContainerStarted","Data":"b46fb3124db1dc141dbaf988d9fbfef68abef0f943c5239a4023030d06c33913"} Jul 03 14:09:53.598141 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:53.597735 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:09:53.598910 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:53.598874 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9777d88-5q287" event={"ID":"83951aa5-0627-4ece-829a-91da7f3545e5","Type":"ContainerStarted","Data":"43b6f0be6484bc7b404c4e102fab5ee94316638b87906bd003ceb6179d28e0b8"} Jul 03 14:09:53.600150 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:53.600126 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xn7tc" event={"ID":"21624337-43ef-4f45-8118-63b9de2f9ed9","Type":"ContainerStarted","Data":"45f2c4e799ee073cab6195ca1cee3d956044129ce0ad0771014db09fd4376f55"} Jul 03 14:09:53.624195 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:53.624144 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" podStartSLOduration=2.624128514 podStartE2EDuration="2.624128514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:53.621624848 +0000 UTC m=+71.977909473" watchObservedRunningTime="2026-07-03 14:09:53.624128514 +0000 UTC m=+71.980413134" Jul 03 14:09:53.644850 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:53.644801 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xn7tc" podStartSLOduration=67.336084201 podStartE2EDuration="1m11.644788025s" totalImagesPullingTime="4.308703824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 14:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:53.644003448 +0000 UTC m=+72.000288073" watchObservedRunningTime="2026-07-03 14:09:53.644788025 +0000 UTC m=+72.001072644" Jul 03 14:09:53.724247 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:53.723885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:53.730059 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:53.730025 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9da1fd-c331-406d-a767-5b50b53f696f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v5kvh\" (UID: \"cc9da1fd-c331-406d-a767-5b50b53f696f\") " pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:53.959579 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:53.959494 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kwmd4\"" Jul 03 14:09:53.966632 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:53.966603 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v5kvh" Jul 03 14:09:54.144105 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:54.144000 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v5kvh"] Jul 03 14:09:54.147372 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:09:54.147340 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9da1fd_c331_406d_a767_5b50b53f696f.slice/crio-70e305435072c2f72ca91dd0b8b7139aa1d37e607e2d7a88c256705881332c23 WatchSource:0}: Error finding container 70e305435072c2f72ca91dd0b8b7139aa1d37e607e2d7a88c256705881332c23: Status 404 returned error can't find the container with id 70e305435072c2f72ca91dd0b8b7139aa1d37e607e2d7a88c256705881332c23 Jul 03 14:09:54.608110 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:54.607999 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v5kvh" event={"ID":"cc9da1fd-c331-406d-a767-5b50b53f696f","Type":"ContainerStarted","Data":"70e305435072c2f72ca91dd0b8b7139aa1d37e607e2d7a88c256705881332c23"} Jul 03 14:09:56.617193 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:56.617158 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9777d88-5q287" event={"ID":"83951aa5-0627-4ece-829a-91da7f3545e5","Type":"ContainerStarted","Data":"9168ab4d06477dffbaacc19f5cec95bfb8c0de84fb2e38b919b9f658459de9a7"} Jul 03 14:09:56.645325 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:56.645260 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9777d88-5q287" podStartSLOduration=3.009250662 podStartE2EDuration="5.645242343s" totalImagesPullingTime="2.635991681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:09:56.644465109 +0000 UTC m=+75.000749730" watchObservedRunningTime="2026-07-03 14:09:56.645242343 +0000 UTC m=+75.001526963" Jul 03 14:09:58.385367 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.385332 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-db848f499-7s52t"] Jul 03 14:09:58.395172 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.395145 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.399862 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.399626 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-svd8d\"" Jul 03 14:09:58.399862 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.399706 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 03 14:09:58.399862 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.399732 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 03 14:09:58.399862 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.399642 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 03 14:09:58.399862 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.399894 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 03 14:09:58.399862 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.399839 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 03 14:09:58.400476 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.400250 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 03 14:09:58.401096 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.401076 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 03 14:09:58.404334 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.404309 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-db848f499-7s52t"] Jul 03 14:09:58.406147 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.406129 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 03 14:09:58.466365 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.466324 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-console-config\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.466528 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.466375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e74b4f16-0f3e-4e91-9815-bff76ca0e191-console-oauth-config\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.466528 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.466428 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bljcq\" (UniqueName: \"kubernetes.io/projected/e74b4f16-0f3e-4e91-9815-bff76ca0e191-kube-api-access-bljcq\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.466635 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.466529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-oauth-serving-cert\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.466635 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.466555 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-trusted-ca-bundle\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.466635 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.466590 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e74b4f16-0f3e-4e91-9815-bff76ca0e191-console-serving-cert\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.466740 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.466685 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-service-ca\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.567991 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.567947 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e74b4f16-0f3e-4e91-9815-bff76ca0e191-console-serving-cert\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.568158 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.568014 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-service-ca\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.568158 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.568063 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-console-config\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.568158 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.568095 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e74b4f16-0f3e-4e91-9815-bff76ca0e191-console-oauth-config\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.568364 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.568303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bljcq\" (UniqueName: \"kubernetes.io/projected/e74b4f16-0f3e-4e91-9815-bff76ca0e191-kube-api-access-bljcq\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.568423 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.568368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-oauth-serving-cert\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.568423 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.568399 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-trusted-ca-bundle\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.568884 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.568861 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-console-config\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.569137 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.569114 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-service-ca\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.569209 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.569192 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-trusted-ca-bundle\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.569344 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.569313 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-oauth-serving-cert\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.570812 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.570791 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e74b4f16-0f3e-4e91-9815-bff76ca0e191-console-serving-cert\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.570898 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.570848 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e74b4f16-0f3e-4e91-9815-bff76ca0e191-console-oauth-config\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.580410 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.580387 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bljcq\" (UniqueName: \"kubernetes.io/projected/e74b4f16-0f3e-4e91-9815-bff76ca0e191-kube-api-access-bljcq\") pod \"console-db848f499-7s52t\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.706226 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.706137 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db848f499-7s52t" Jul 03 14:09:58.907174 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:58.907042 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-db848f499-7s52t"] Jul 03 14:09:58.910893 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:09:58.910856 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74b4f16_0f3e_4e91_9815_bff76ca0e191.slice/crio-67975e1bc9c7d3c5c73784054d907cddbb44b5fee6ba88b1a271ae4e6afc16e3 WatchSource:0}: Error finding container 67975e1bc9c7d3c5c73784054d907cddbb44b5fee6ba88b1a271ae4e6afc16e3: Status 404 returned error can't find the container with id 67975e1bc9c7d3c5c73784054d907cddbb44b5fee6ba88b1a271ae4e6afc16e3 Jul 03 14:09:59.633526 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:59.633427 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v5kvh" event={"ID":"cc9da1fd-c331-406d-a767-5b50b53f696f","Type":"ContainerStarted","Data":"d71ef9c45a01c8cd0fda7d7987a3bb0092ebf345b4614b7a59536307b09a53f9"} Jul 03 14:09:59.633526 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:59.633505 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v5kvh" event={"ID":"cc9da1fd-c331-406d-a767-5b50b53f696f","Type":"ContainerStarted","Data":"6546d080af4c6d1bb70be434a572dbc5dce756c53965facf4ba74a331bbc09b7"} Jul 03 14:09:59.635307 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:09:59.635272 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db848f499-7s52t" event={"ID":"e74b4f16-0f3e-4e91-9815-bff76ca0e191","Type":"ContainerStarted","Data":"67975e1bc9c7d3c5c73784054d907cddbb44b5fee6ba88b1a271ae4e6afc16e3"} Jul 03 14:10:01.346645 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.346611 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9"] Jul 03 14:10:01.352286 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.352259 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9" Jul 03 14:10:01.357047 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.357018 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 03 14:10:01.357047 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.357041 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 03 14:10:01.359220 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.359198 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8zxs5\"" Jul 03 14:10:01.359380 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.359295 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 03 14:10:01.362031 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.361995 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 03 14:10:01.362281 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.362261 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 03 14:10:01.375938 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.375898 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9"] Jul 03 14:10:01.491635 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.491589 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/398622f5-0feb-4a05-a51d-64ff760605ca-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-tslg9\" (UID: \"398622f5-0feb-4a05-a51d-64ff760605ca\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9" Jul 03 14:10:01.491846 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.491781 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98wvh\" (UniqueName: \"kubernetes.io/projected/398622f5-0feb-4a05-a51d-64ff760605ca-kube-api-access-98wvh\") pod \"prometheus-operator-567f7b8b7c-tslg9\" (UID: \"398622f5-0feb-4a05-a51d-64ff760605ca\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9" Jul 03 14:10:01.491846 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.491825 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/398622f5-0feb-4a05-a51d-64ff760605ca-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-tslg9\" (UID: \"398622f5-0feb-4a05-a51d-64ff760605ca\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9" Jul 03 14:10:01.491956 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.491851 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/398622f5-0feb-4a05-a51d-64ff760605ca-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-tslg9\" (UID: \"398622f5-0feb-4a05-a51d-64ff760605ca\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9" Jul 03 14:10:01.593317 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.593278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98wvh\" (UniqueName: \"kubernetes.io/projected/398622f5-0feb-4a05-a51d-64ff760605ca-kube-api-access-98wvh\") pod \"prometheus-operator-567f7b8b7c-tslg9\" (UID: \"398622f5-0feb-4a05-a51d-64ff760605ca\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9" Jul 03 14:10:01.593515 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.593333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/398622f5-0feb-4a05-a51d-64ff760605ca-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-tslg9\" (UID: \"398622f5-0feb-4a05-a51d-64ff760605ca\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9" Jul 03 14:10:01.593515 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.593358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/398622f5-0feb-4a05-a51d-64ff760605ca-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-tslg9\" (UID: \"398622f5-0feb-4a05-a51d-64ff760605ca\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9" Jul 03 14:10:01.593515 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.593407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/398622f5-0feb-4a05-a51d-64ff760605ca-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-tslg9\" (UID: \"398622f5-0feb-4a05-a51d-64ff760605ca\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9" Jul 03 14:10:01.594145 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.594114 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/398622f5-0feb-4a05-a51d-64ff760605ca-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-tslg9\" (UID: \"398622f5-0feb-4a05-a51d-64ff760605ca\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9" Jul 03 14:10:01.596337 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.596302 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/398622f5-0feb-4a05-a51d-64ff760605ca-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-tslg9\" (UID: \"398622f5-0feb-4a05-a51d-64ff760605ca\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9" Jul 03 14:10:01.596473 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.596411 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/398622f5-0feb-4a05-a51d-64ff760605ca-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-tslg9\" (UID: \"398622f5-0feb-4a05-a51d-64ff760605ca\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9" Jul 03 14:10:01.603949 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.603882 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98wvh\" (UniqueName: \"kubernetes.io/projected/398622f5-0feb-4a05-a51d-64ff760605ca-kube-api-access-98wvh\") pod \"prometheus-operator-567f7b8b7c-tslg9\" (UID: \"398622f5-0feb-4a05-a51d-64ff760605ca\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9" Jul 03 14:10:01.668153 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:01.668112 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9" Jul 03 14:10:02.602922 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:02.602895 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wx4zj" Jul 03 14:10:03.133277 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:03.133237 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9"] Jul 03 14:10:03.136251 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:10:03.136218 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398622f5_0feb_4a05_a51d_64ff760605ca.slice/crio-95c8e8ecaa6267dfa1cce8f2b139b4d29ce645f0b66e9c9384237e3b9a79e7c7 WatchSource:0}: Error finding container 95c8e8ecaa6267dfa1cce8f2b139b4d29ce645f0b66e9c9384237e3b9a79e7c7: Status 404 returned error can't find the container with id 95c8e8ecaa6267dfa1cce8f2b139b4d29ce645f0b66e9c9384237e3b9a79e7c7 Jul 03 14:10:03.653514 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:03.653469 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db848f499-7s52t" event={"ID":"e74b4f16-0f3e-4e91-9815-bff76ca0e191","Type":"ContainerStarted","Data":"dbee81de16953dee5a878f78acf2a236149d4193f5d89a3c7e6fb6f2f5ed6072"} Jul 03 14:10:03.656153 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:03.656121 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v5kvh" event={"ID":"cc9da1fd-c331-406d-a767-5b50b53f696f","Type":"ContainerStarted","Data":"dc4cd2bcbab170f3a902dbe0087caaea719e09f8de740076aa3a5d31cae50d9a"} Jul 03 14:10:03.657430 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:03.657405 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9" event={"ID":"398622f5-0feb-4a05-a51d-64ff760605ca","Type":"ContainerStarted","Data":"95c8e8ecaa6267dfa1cce8f2b139b4d29ce645f0b66e9c9384237e3b9a79e7c7"} Jul 03 14:10:03.674239 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:03.674163 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-db848f499-7s52t" podStartSLOduration=1.5844021929999998 podStartE2EDuration="5.674143958s" totalImagesPullingTime="4.089741765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:10:03.674001378 +0000 UTC m=+82.030285998" watchObservedRunningTime="2026-07-03 14:10:03.674143958 +0000 UTC m=+82.030428578" Jul 03 14:10:03.694882 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:03.694802 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v5kvh" podStartSLOduration=33.962813409 podStartE2EDuration="42.694785844s" totalImagesPullingTime="8.731972435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 14:09:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:10:03.694596684 +0000 UTC m=+82.050881301" watchObservedRunningTime="2026-07-03 14:10:03.694785844 +0000 UTC m=+82.051070506" Jul 03 14:10:04.664880 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:04.664828 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9" event={"ID":"398622f5-0feb-4a05-a51d-64ff760605ca","Type":"ContainerStarted","Data":"0a40b48e0f7228489482cb8fdcb6079dca7946ac110fed046db23f8f8e64c64b"} Jul 03 14:10:04.665382 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:04.664904 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9" event={"ID":"398622f5-0feb-4a05-a51d-64ff760605ca","Type":"ContainerStarted","Data":"8ea2d22bdb5a529b6232fca8c6cdbee50d9099596b917229a30ce10ef29c5645"} Jul 03 14:10:04.685610 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:04.685556 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-tslg9" podStartSLOduration=2.3854749650000002 podStartE2EDuration="3.685540415s" totalImagesPullingTime="1.30006545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:10:04.684034008 +0000 UTC m=+83.040318631" watchObservedRunningTime="2026-07-03 14:10:04.685540415 +0000 UTC m=+83.041825064" Jul 03 14:10:06.789535 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.789504 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz"] Jul 03 14:10:06.793545 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.793527 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:06.796480 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.796458 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 03 14:10:06.796690 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.796649 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-c5t8z\"" Jul 03 14:10:06.798128 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.798102 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 03 14:10:06.798284 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.798268 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 03 14:10:06.803680 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.803638 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz"] Jul 03 14:10:06.811599 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.811576 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-97nhz"] Jul 03 14:10:06.813920 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.813905 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:06.817943 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.817645 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 03 14:10:06.817943 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.817738 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-54cgl\"" Jul 03 14:10:06.818085 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.818009 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 03 14:10:06.818085 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.818045 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 03 14:10:06.936791 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.936761 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5d47a63-27a9-4aa2-9e30-0c031a2da741-node-exporter-textfile\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:06.937054 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.936795 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5d47a63-27a9-4aa2-9e30-0c031a2da741-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:06.937054 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.936828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7qxt\" (UniqueName: \"kubernetes.io/projected/87429e78-2975-4107-8cd7-aebcd695ad4f-kube-api-access-h7qxt\") pod \"kube-state-metrics-7c9dbb8b7d-b4gdz\" (UID: \"87429e78-2975-4107-8cd7-aebcd695ad4f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:06.937490 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.937306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87429e78-2975-4107-8cd7-aebcd695ad4f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-b4gdz\" (UID: \"87429e78-2975-4107-8cd7-aebcd695ad4f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:06.937490 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.937394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5d47a63-27a9-4aa2-9e30-0c031a2da741-root\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:06.942579 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.937688 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87429e78-2975-4107-8cd7-aebcd695ad4f-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-b4gdz\" (UID: \"87429e78-2975-4107-8cd7-aebcd695ad4f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:06.942579 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.937743 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d47a63-27a9-4aa2-9e30-0c031a2da741-sys\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:06.942579 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.937781 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5d47a63-27a9-4aa2-9e30-0c031a2da741-node-exporter-accelerators-collector-config\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:06.942579 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.937827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5d47a63-27a9-4aa2-9e30-0c031a2da741-node-exporter-tls\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:06.942579 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.937859 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5d47a63-27a9-4aa2-9e30-0c031a2da741-metrics-client-ca\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:06.942579 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.937903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5d47a63-27a9-4aa2-9e30-0c031a2da741-node-exporter-wtmp\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:06.942579 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.937931 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssf78\" (UniqueName: \"kubernetes.io/projected/b5d47a63-27a9-4aa2-9e30-0c031a2da741-kube-api-access-ssf78\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:06.942579 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.937962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/87429e78-2975-4107-8cd7-aebcd695ad4f-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-b4gdz\" (UID: \"87429e78-2975-4107-8cd7-aebcd695ad4f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:06.942579 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.937990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87429e78-2975-4107-8cd7-aebcd695ad4f-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-b4gdz\" (UID: \"87429e78-2975-4107-8cd7-aebcd695ad4f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:06.942579 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:06.938032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/87429e78-2975-4107-8cd7-aebcd695ad4f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-b4gdz\" (UID: \"87429e78-2975-4107-8cd7-aebcd695ad4f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:07.039287 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.039248 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87429e78-2975-4107-8cd7-aebcd695ad4f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-b4gdz\" (UID: \"87429e78-2975-4107-8cd7-aebcd695ad4f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:07.039476 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.039296 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5d47a63-27a9-4aa2-9e30-0c031a2da741-root\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:07.039476 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.039323 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87429e78-2975-4107-8cd7-aebcd695ad4f-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-b4gdz\" (UID: \"87429e78-2975-4107-8cd7-aebcd695ad4f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:07.039476 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.039353 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d47a63-27a9-4aa2-9e30-0c031a2da741-sys\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:07.039476 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.039382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5d47a63-27a9-4aa2-9e30-0c031a2da741-node-exporter-accelerators-collector-config\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:07.039476 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.039424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5d47a63-27a9-4aa2-9e30-0c031a2da741-node-exporter-tls\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:07.039476 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.039451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5d47a63-27a9-4aa2-9e30-0c031a2da741-metrics-client-ca\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:07.039822 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.039486 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5d47a63-27a9-4aa2-9e30-0c031a2da741-node-exporter-wtmp\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:07.039822 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.039512 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssf78\" (UniqueName: \"kubernetes.io/projected/b5d47a63-27a9-4aa2-9e30-0c031a2da741-kube-api-access-ssf78\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:07.039822 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.039533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/87429e78-2975-4107-8cd7-aebcd695ad4f-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-b4gdz\" (UID: \"87429e78-2975-4107-8cd7-aebcd695ad4f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:07.039822 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:10:07.039543 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 03 14:10:07.039822 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.039560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87429e78-2975-4107-8cd7-aebcd695ad4f-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-b4gdz\" (UID: \"87429e78-2975-4107-8cd7-aebcd695ad4f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:07.039822 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.039597 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/87429e78-2975-4107-8cd7-aebcd695ad4f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-b4gdz\" (UID: \"87429e78-2975-4107-8cd7-aebcd695ad4f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:07.039822 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:10:07.039625 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5d47a63-27a9-4aa2-9e30-0c031a2da741-node-exporter-tls podName:b5d47a63-27a9-4aa2-9e30-0c031a2da741 nodeName:}" failed. No retries permitted until 2026-07-03 14:10:07.539602329 +0000 UTC m=+85.895886941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b5d47a63-27a9-4aa2-9e30-0c031a2da741-node-exporter-tls") pod "node-exporter-97nhz" (UID: "b5d47a63-27a9-4aa2-9e30-0c031a2da741") : secret "node-exporter-tls" not found Jul 03 14:10:07.039822 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.039696 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5d47a63-27a9-4aa2-9e30-0c031a2da741-node-exporter-textfile\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:07.039822 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.039724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5d47a63-27a9-4aa2-9e30-0c031a2da741-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:07.039822 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.039750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7qxt\" (UniqueName: \"kubernetes.io/projected/87429e78-2975-4107-8cd7-aebcd695ad4f-kube-api-access-h7qxt\") pod \"kube-state-metrics-7c9dbb8b7d-b4gdz\" (UID: \"87429e78-2975-4107-8cd7-aebcd695ad4f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:07.040301 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.040009 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5d47a63-27a9-4aa2-9e30-0c031a2da741-node-exporter-accelerators-collector-config\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:07.040301 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.040145 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5d47a63-27a9-4aa2-9e30-0c031a2da741-node-exporter-textfile\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:07.040301 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.040152 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5d47a63-27a9-4aa2-9e30-0c031a2da741-node-exporter-wtmp\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:07.040301 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.039419 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5d47a63-27a9-4aa2-9e30-0c031a2da741-root\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:07.040301 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.040294 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/87429e78-2975-4107-8cd7-aebcd695ad4f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-b4gdz\" (UID: \"87429e78-2975-4107-8cd7-aebcd695ad4f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:07.040593 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.040571 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5d47a63-27a9-4aa2-9e30-0c031a2da741-metrics-client-ca\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:07.040714 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.040687 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/87429e78-2975-4107-8cd7-aebcd695ad4f-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-b4gdz\" (UID: \"87429e78-2975-4107-8cd7-aebcd695ad4f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:07.040820 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.040780 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d47a63-27a9-4aa2-9e30-0c031a2da741-sys\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:07.041605 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.041587 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87429e78-2975-4107-8cd7-aebcd695ad4f-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-b4gdz\" (UID: \"87429e78-2975-4107-8cd7-aebcd695ad4f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:07.043023 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.042970 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87429e78-2975-4107-8cd7-aebcd695ad4f-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-b4gdz\" (UID: \"87429e78-2975-4107-8cd7-aebcd695ad4f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:07.043198 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.043178 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5d47a63-27a9-4aa2-9e30-0c031a2da741-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:07.043463 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.043443 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87429e78-2975-4107-8cd7-aebcd695ad4f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-b4gdz\" (UID: \"87429e78-2975-4107-8cd7-aebcd695ad4f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:07.050738 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.050717 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssf78\" (UniqueName: \"kubernetes.io/projected/b5d47a63-27a9-4aa2-9e30-0c031a2da741-kube-api-access-ssf78\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:07.051067 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.051048 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7qxt\" (UniqueName: \"kubernetes.io/projected/87429e78-2975-4107-8cd7-aebcd695ad4f-kube-api-access-h7qxt\") pod \"kube-state-metrics-7c9dbb8b7d-b4gdz\" (UID: \"87429e78-2975-4107-8cd7-aebcd695ad4f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:07.105039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.105009 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" Jul 03 14:10:07.254182 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.254151 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz"] Jul 03 14:10:07.257963 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:10:07.257939 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87429e78_2975_4107_8cd7_aebcd695ad4f.slice/crio-c920504596b162c45dd3cdce4e609a5d97d895b57f8ced54db6e0e2e86e9dc6b WatchSource:0}: Error finding container c920504596b162c45dd3cdce4e609a5d97d895b57f8ced54db6e0e2e86e9dc6b: Status 404 returned error can't find the container with id c920504596b162c45dd3cdce4e609a5d97d895b57f8ced54db6e0e2e86e9dc6b Jul 03 14:10:07.545983 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.545949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5d47a63-27a9-4aa2-9e30-0c031a2da741-node-exporter-tls\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:07.548573 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.548549 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5d47a63-27a9-4aa2-9e30-0c031a2da741-node-exporter-tls\") pod \"node-exporter-97nhz\" (UID: \"b5d47a63-27a9-4aa2-9e30-0c031a2da741\") " pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:07.679102 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.679060 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" event={"ID":"87429e78-2975-4107-8cd7-aebcd695ad4f","Type":"ContainerStarted","Data":"c920504596b162c45dd3cdce4e609a5d97d895b57f8ced54db6e0e2e86e9dc6b"} Jul 03 14:10:07.724165 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:07.724095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-97nhz" Jul 03 14:10:08.684646 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:08.684603 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-97nhz" event={"ID":"b5d47a63-27a9-4aa2-9e30-0c031a2da741","Type":"ContainerStarted","Data":"3e35a3ac42bdb14a38e026ac0a981bf03662203f7cb46784ab3f0b9e2c35571d"} Jul 03 14:10:08.707319 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:08.707285 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-db848f499-7s52t" Jul 03 14:10:08.707505 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:08.707363 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-db848f499-7s52t" Jul 03 14:10:08.713175 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:08.713151 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-db848f499-7s52t" Jul 03 14:10:09.695725 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:09.695682 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" event={"ID":"87429e78-2975-4107-8cd7-aebcd695ad4f","Type":"ContainerStarted","Data":"52cc93b53cd340540207fc00999ffc793652352766554185f285839f44a45064"} Jul 03 14:10:09.696171 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:09.695754 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" event={"ID":"87429e78-2975-4107-8cd7-aebcd695ad4f","Type":"ContainerStarted","Data":"fb25b628a25b8ad6f0fc884d776f4e9982637bf01af172a2bd4a8d4db0df9d16"} Jul 03 14:10:09.696171 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:09.695784 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" event={"ID":"87429e78-2975-4107-8cd7-aebcd695ad4f","Type":"ContainerStarted","Data":"827ce128cae78c151cef2734c801a3f205192715487a78238284730147e36d8f"} Jul 03 14:10:09.697756 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:09.697729 2576 generic.go:356] "Generic (PLEG): container finished" podID="b5d47a63-27a9-4aa2-9e30-0c031a2da741" containerID="6253c7c7a2fcb35cefc4baadb2dba9d8713915d8d8cfd4ac779d754ab799d712" exitCode=0 Jul 03 14:10:09.697882 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:09.697828 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-97nhz" event={"ID":"b5d47a63-27a9-4aa2-9e30-0c031a2da741","Type":"ContainerDied","Data":"6253c7c7a2fcb35cefc4baadb2dba9d8713915d8d8cfd4ac779d754ab799d712"} Jul 03 14:10:09.702382 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:09.702364 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-db848f499-7s52t" Jul 03 14:10:09.758872 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:09.758815 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b4gdz" podStartSLOduration=2.239385202 podStartE2EDuration="3.758750259s" totalImagesPullingTime="1.519365057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:10:09.757783025 +0000 UTC m=+88.114067647" watchObservedRunningTime="2026-07-03 14:10:09.758750259 +0000 UTC m=+88.115034881" Jul 03 14:10:10.703621 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:10.703579 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-97nhz" event={"ID":"b5d47a63-27a9-4aa2-9e30-0c031a2da741","Type":"ContainerStarted","Data":"7a80334e96d88647871f1da762cdb3f8b218238267b134178ff0fce37c035779"} Jul 03 14:10:10.704015 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:10.703638 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-97nhz" event={"ID":"b5d47a63-27a9-4aa2-9e30-0c031a2da741","Type":"ContainerStarted","Data":"e5c27270010bd75b8796663b81670e661685e20fefb0c8fa1ef73615c7894468"} Jul 03 14:10:10.755117 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:10.755054 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-97nhz" podStartSLOduration=3.611548048 podStartE2EDuration="4.755040087s" totalImagesPullingTime="1.043381408s" totalInitContainerRuntime="100.110631ms" isStatefulPod=true podCreationTimestamp="2026-07-03 14:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:10:10.753348739 +0000 UTC m=+89.109633372" watchObservedRunningTime="2026-07-03 14:10:10.755040087 +0000 UTC m=+89.111324707" Jul 03 14:10:14.613031 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:14.613000 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d699c5c4-fv99h" Jul 03 14:10:19.577635 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:19.577597 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6n5dr" Jul 03 14:10:24.819890 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:24.819852 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-db848f499-7s52t"] Jul 03 14:10:49.840471 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:49.840403 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-db848f499-7s52t" podUID="e74b4f16-0f3e-4e91-9815-bff76ca0e191" containerName="console" containerID="cri-o://dbee81de16953dee5a878f78acf2a236149d4193f5d89a3c7e6fb6f2f5ed6072" gracePeriod=15 Jul 03 14:10:50.080814 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.080790 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db848f499-7s52t" Jul 03 14:10:50.168760 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.168686 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-console-config\") pod \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " Jul 03 14:10:50.168760 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.168735 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-oauth-serving-cert\") pod \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " Jul 03 14:10:50.168967 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.168776 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e74b4f16-0f3e-4e91-9815-bff76ca0e191-console-oauth-config\") pod \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " Jul 03 14:10:50.168967 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.168824 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e74b4f16-0f3e-4e91-9815-bff76ca0e191-console-serving-cert\") pod \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " Jul 03 14:10:50.168967 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.168882 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-service-ca\") pod \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " Jul 03 14:10:50.168967 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.168923 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bljcq\" (UniqueName: \"kubernetes.io/projected/e74b4f16-0f3e-4e91-9815-bff76ca0e191-kube-api-access-bljcq\") pod \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " Jul 03 14:10:50.168967 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.168949 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-trusted-ca-bundle\") pod \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\" (UID: \"e74b4f16-0f3e-4e91-9815-bff76ca0e191\") " Jul 03 14:10:50.169208 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.169018 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-console-config" (OuterVolumeSpecName: "console-config") pod "e74b4f16-0f3e-4e91-9815-bff76ca0e191" (UID: "e74b4f16-0f3e-4e91-9815-bff76ca0e191"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:10:50.169274 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.169116 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e74b4f16-0f3e-4e91-9815-bff76ca0e191" (UID: "e74b4f16-0f3e-4e91-9815-bff76ca0e191"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:10:50.169325 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.169294 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-console-config\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:10:50.169325 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.169302 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-service-ca" (OuterVolumeSpecName: "service-ca") pod "e74b4f16-0f3e-4e91-9815-bff76ca0e191" (UID: "e74b4f16-0f3e-4e91-9815-bff76ca0e191"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:10:50.169419 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.169312 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-oauth-serving-cert\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:10:50.169419 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.169352 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e74b4f16-0f3e-4e91-9815-bff76ca0e191" (UID: "e74b4f16-0f3e-4e91-9815-bff76ca0e191"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:10:50.171166 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.171145 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e74b4f16-0f3e-4e91-9815-bff76ca0e191-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e74b4f16-0f3e-4e91-9815-bff76ca0e191" (UID: "e74b4f16-0f3e-4e91-9815-bff76ca0e191"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 14:10:50.171673 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.171624 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e74b4f16-0f3e-4e91-9815-bff76ca0e191-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e74b4f16-0f3e-4e91-9815-bff76ca0e191" (UID: "e74b4f16-0f3e-4e91-9815-bff76ca0e191"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 14:10:50.171673 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.171634 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74b4f16-0f3e-4e91-9815-bff76ca0e191-kube-api-access-bljcq" (OuterVolumeSpecName: "kube-api-access-bljcq") pod "e74b4f16-0f3e-4e91-9815-bff76ca0e191" (UID: "e74b4f16-0f3e-4e91-9815-bff76ca0e191"). InnerVolumeSpecName "kube-api-access-bljcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:10:50.269718 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.269681 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bljcq\" (UniqueName: \"kubernetes.io/projected/e74b4f16-0f3e-4e91-9815-bff76ca0e191-kube-api-access-bljcq\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:10:50.269718 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.269714 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-trusted-ca-bundle\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:10:50.269718 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.269724 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e74b4f16-0f3e-4e91-9815-bff76ca0e191-console-oauth-config\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:10:50.269938 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.269732 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e74b4f16-0f3e-4e91-9815-bff76ca0e191-console-serving-cert\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:10:50.269938 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.269741 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e74b4f16-0f3e-4e91-9815-bff76ca0e191-service-ca\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:10:50.853107 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.853073 2576 generic.go:356] "Generic (PLEG): container finished" podID="e74b4f16-0f3e-4e91-9815-bff76ca0e191" containerID="dbee81de16953dee5a878f78acf2a236149d4193f5d89a3c7e6fb6f2f5ed6072" exitCode=0 Jul 03 14:10:50.853577 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.853120 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db848f499-7s52t" Jul 03 14:10:50.853577 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.853165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db848f499-7s52t" event={"ID":"e74b4f16-0f3e-4e91-9815-bff76ca0e191","Type":"ContainerDied","Data":"dbee81de16953dee5a878f78acf2a236149d4193f5d89a3c7e6fb6f2f5ed6072"} Jul 03 14:10:50.853577 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.853220 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db848f499-7s52t" event={"ID":"e74b4f16-0f3e-4e91-9815-bff76ca0e191","Type":"ContainerDied","Data":"67975e1bc9c7d3c5c73784054d907cddbb44b5fee6ba88b1a271ae4e6afc16e3"} Jul 03 14:10:50.853577 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.853243 2576 scope.go:117] "RemoveContainer" containerID="dbee81de16953dee5a878f78acf2a236149d4193f5d89a3c7e6fb6f2f5ed6072" Jul 03 14:10:50.864022 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.863953 2576 scope.go:117] "RemoveContainer" containerID="dbee81de16953dee5a878f78acf2a236149d4193f5d89a3c7e6fb6f2f5ed6072" Jul 03 14:10:50.864521 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:10:50.864494 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbee81de16953dee5a878f78acf2a236149d4193f5d89a3c7e6fb6f2f5ed6072\": container with ID starting with dbee81de16953dee5a878f78acf2a236149d4193f5d89a3c7e6fb6f2f5ed6072 not found: ID does not exist" containerID="dbee81de16953dee5a878f78acf2a236149d4193f5d89a3c7e6fb6f2f5ed6072" Jul 03 14:10:50.864579 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.864535 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbee81de16953dee5a878f78acf2a236149d4193f5d89a3c7e6fb6f2f5ed6072"} err="failed to get container status \"dbee81de16953dee5a878f78acf2a236149d4193f5d89a3c7e6fb6f2f5ed6072\": rpc error: code = NotFound desc = could not find container \"dbee81de16953dee5a878f78acf2a236149d4193f5d89a3c7e6fb6f2f5ed6072\": container with ID starting with dbee81de16953dee5a878f78acf2a236149d4193f5d89a3c7e6fb6f2f5ed6072 not found: ID does not exist" Jul 03 14:10:50.887515 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.887490 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-db848f499-7s52t"] Jul 03 14:10:50.891369 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:50.891348 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-db848f499-7s52t"] Jul 03 14:10:52.167317 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:10:52.167286 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e74b4f16-0f3e-4e91-9815-bff76ca0e191" path="/var/lib/kubelet/pods/e74b4f16-0f3e-4e91-9815-bff76ca0e191/volumes" Jul 03 14:13:42.041162 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:13:42.041132 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:13:42.044511 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:13:42.044481 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:13:42.045210 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:13:42.045187 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:13:42.048780 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:13:42.048762 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:13:42.051799 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:13:42.051781 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 03 14:14:15.550987 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:15.550948 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd"] Jul 03 14:14:15.552497 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:15.551216 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e74b4f16-0f3e-4e91-9815-bff76ca0e191" containerName="console" Jul 03 14:14:15.552497 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:15.551227 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e74b4f16-0f3e-4e91-9815-bff76ca0e191" containerName="console" Jul 03 14:14:15.552497 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:15.551277 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e74b4f16-0f3e-4e91-9815-bff76ca0e191" containerName="console" Jul 03 14:14:15.553337 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:15.553322 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd" Jul 03 14:14:15.556238 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:15.556219 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 03 14:14:15.557599 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:15.557575 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-d7mkw\"" Jul 03 14:14:15.557715 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:15.557582 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 03 14:14:15.568300 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:15.568276 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd"] Jul 03 14:14:15.622743 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:15.622710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/822ffb1e-c4be-4af0-ae0a-2e06562b918e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd\" (UID: \"822ffb1e-c4be-4af0-ae0a-2e06562b918e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd" Jul 03 14:14:15.622743 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:15.622745 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/822ffb1e-c4be-4af0-ae0a-2e06562b918e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd\" (UID: \"822ffb1e-c4be-4af0-ae0a-2e06562b918e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd" Jul 03 14:14:15.622940 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:15.622788 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spnmp\" (UniqueName: \"kubernetes.io/projected/822ffb1e-c4be-4af0-ae0a-2e06562b918e-kube-api-access-spnmp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd\" (UID: \"822ffb1e-c4be-4af0-ae0a-2e06562b918e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd" Jul 03 14:14:15.724074 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:15.724045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/822ffb1e-c4be-4af0-ae0a-2e06562b918e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd\" (UID: \"822ffb1e-c4be-4af0-ae0a-2e06562b918e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd" Jul 03 14:14:15.724233 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:15.724081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/822ffb1e-c4be-4af0-ae0a-2e06562b918e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd\" (UID: \"822ffb1e-c4be-4af0-ae0a-2e06562b918e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd" Jul 03 14:14:15.724233 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:15.724130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spnmp\" (UniqueName: \"kubernetes.io/projected/822ffb1e-c4be-4af0-ae0a-2e06562b918e-kube-api-access-spnmp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd\" (UID: \"822ffb1e-c4be-4af0-ae0a-2e06562b918e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd" Jul 03 14:14:15.724427 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:15.724406 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/822ffb1e-c4be-4af0-ae0a-2e06562b918e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd\" (UID: \"822ffb1e-c4be-4af0-ae0a-2e06562b918e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd" Jul 03 14:14:15.724465 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:15.724431 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/822ffb1e-c4be-4af0-ae0a-2e06562b918e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd\" (UID: \"822ffb1e-c4be-4af0-ae0a-2e06562b918e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd" Jul 03 14:14:15.734388 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:15.734363 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spnmp\" (UniqueName: \"kubernetes.io/projected/822ffb1e-c4be-4af0-ae0a-2e06562b918e-kube-api-access-spnmp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd\" (UID: \"822ffb1e-c4be-4af0-ae0a-2e06562b918e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd" Jul 03 14:14:15.862636 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:15.862551 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd" Jul 03 14:14:16.001290 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:16.001264 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd"] Jul 03 14:14:16.002930 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:14:16.002902 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822ffb1e_c4be_4af0_ae0a_2e06562b918e.slice/crio-1ae2435ead91062cef4e0fade6cb2d21f8148615d570bbdfffc72c16bd95509a WatchSource:0}: Error finding container 1ae2435ead91062cef4e0fade6cb2d21f8148615d570bbdfffc72c16bd95509a: Status 404 returned error can't find the container with id 1ae2435ead91062cef4e0fade6cb2d21f8148615d570bbdfffc72c16bd95509a Jul 03 14:14:16.004964 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:16.004949 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 03 14:14:16.600364 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:16.600325 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd" event={"ID":"822ffb1e-c4be-4af0-ae0a-2e06562b918e","Type":"ContainerStarted","Data":"1ae2435ead91062cef4e0fade6cb2d21f8148615d570bbdfffc72c16bd95509a"} Jul 03 14:14:21.621673 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:21.621627 2576 generic.go:356] "Generic (PLEG): container finished" podID="822ffb1e-c4be-4af0-ae0a-2e06562b918e" containerID="acefa447f52d5a137aa8d2e1f3e2c11fda851c0410b9fbf513e4f75bb503bce7" exitCode=0 Jul 03 14:14:21.621984 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:21.621700 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd" event={"ID":"822ffb1e-c4be-4af0-ae0a-2e06562b918e","Type":"ContainerDied","Data":"acefa447f52d5a137aa8d2e1f3e2c11fda851c0410b9fbf513e4f75bb503bce7"} Jul 03 14:14:23.630120 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:23.630034 2576 generic.go:356] "Generic (PLEG): container finished" podID="822ffb1e-c4be-4af0-ae0a-2e06562b918e" containerID="fb646b5338771d9146e6848d37eea338dfdf2901f190a0e6f671afe4a61dd54a" exitCode=0 Jul 03 14:14:23.630120 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:23.630105 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd" event={"ID":"822ffb1e-c4be-4af0-ae0a-2e06562b918e","Type":"ContainerDied","Data":"fb646b5338771d9146e6848d37eea338dfdf2901f190a0e6f671afe4a61dd54a"} Jul 03 14:14:31.662952 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:31.662914 2576 generic.go:356] "Generic (PLEG): container finished" podID="822ffb1e-c4be-4af0-ae0a-2e06562b918e" containerID="aed7efe7f4d9508e673a65aeda5c7328d16de953c33c2586841af0a1dd46d5b5" exitCode=0 Jul 03 14:14:31.663369 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:31.662996 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd" event={"ID":"822ffb1e-c4be-4af0-ae0a-2e06562b918e","Type":"ContainerDied","Data":"aed7efe7f4d9508e673a65aeda5c7328d16de953c33c2586841af0a1dd46d5b5"} Jul 03 14:14:32.793230 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:32.793209 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd" Jul 03 14:14:32.869027 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:32.868999 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/822ffb1e-c4be-4af0-ae0a-2e06562b918e-util\") pod \"822ffb1e-c4be-4af0-ae0a-2e06562b918e\" (UID: \"822ffb1e-c4be-4af0-ae0a-2e06562b918e\") " Jul 03 14:14:32.869221 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:32.869047 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spnmp\" (UniqueName: \"kubernetes.io/projected/822ffb1e-c4be-4af0-ae0a-2e06562b918e-kube-api-access-spnmp\") pod \"822ffb1e-c4be-4af0-ae0a-2e06562b918e\" (UID: \"822ffb1e-c4be-4af0-ae0a-2e06562b918e\") " Jul 03 14:14:32.869221 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:32.869091 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/822ffb1e-c4be-4af0-ae0a-2e06562b918e-bundle\") pod \"822ffb1e-c4be-4af0-ae0a-2e06562b918e\" (UID: \"822ffb1e-c4be-4af0-ae0a-2e06562b918e\") " Jul 03 14:14:32.869844 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:32.869819 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822ffb1e-c4be-4af0-ae0a-2e06562b918e-bundle" (OuterVolumeSpecName: "bundle") pod "822ffb1e-c4be-4af0-ae0a-2e06562b918e" (UID: "822ffb1e-c4be-4af0-ae0a-2e06562b918e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:14:32.871393 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:32.871370 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822ffb1e-c4be-4af0-ae0a-2e06562b918e-kube-api-access-spnmp" (OuterVolumeSpecName: "kube-api-access-spnmp") pod "822ffb1e-c4be-4af0-ae0a-2e06562b918e" (UID: "822ffb1e-c4be-4af0-ae0a-2e06562b918e"). InnerVolumeSpecName "kube-api-access-spnmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:14:32.874015 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:32.873987 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822ffb1e-c4be-4af0-ae0a-2e06562b918e-util" (OuterVolumeSpecName: "util") pod "822ffb1e-c4be-4af0-ae0a-2e06562b918e" (UID: "822ffb1e-c4be-4af0-ae0a-2e06562b918e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:14:32.970284 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:32.970175 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/822ffb1e-c4be-4af0-ae0a-2e06562b918e-bundle\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:14:32.970284 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:32.970221 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/822ffb1e-c4be-4af0-ae0a-2e06562b918e-util\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:14:32.970284 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:32.970249 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spnmp\" (UniqueName: \"kubernetes.io/projected/822ffb1e-c4be-4af0-ae0a-2e06562b918e-kube-api-access-spnmp\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:14:33.671452 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:33.671414 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd" event={"ID":"822ffb1e-c4be-4af0-ae0a-2e06562b918e","Type":"ContainerDied","Data":"1ae2435ead91062cef4e0fade6cb2d21f8148615d570bbdfffc72c16bd95509a"} Jul 03 14:14:33.671452 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:33.671454 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tq4cd" Jul 03 14:14:33.671668 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:33.671464 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ae2435ead91062cef4e0fade6cb2d21f8148615d570bbdfffc72c16bd95509a" Jul 03 14:14:38.543286 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.543250 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-whqcc"] Jul 03 14:14:38.543680 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.543534 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822ffb1e-c4be-4af0-ae0a-2e06562b918e" containerName="util" Jul 03 14:14:38.543680 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.543549 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="822ffb1e-c4be-4af0-ae0a-2e06562b918e" containerName="util" Jul 03 14:14:38.543680 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.543567 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822ffb1e-c4be-4af0-ae0a-2e06562b918e" containerName="extract" Jul 03 14:14:38.543680 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.543574 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="822ffb1e-c4be-4af0-ae0a-2e06562b918e" containerName="extract" Jul 03 14:14:38.543680 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.543584 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822ffb1e-c4be-4af0-ae0a-2e06562b918e" containerName="pull" Jul 03 14:14:38.543680 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.543589 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="822ffb1e-c4be-4af0-ae0a-2e06562b918e" containerName="pull" Jul 03 14:14:38.543680 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.543643 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="822ffb1e-c4be-4af0-ae0a-2e06562b918e" containerName="extract" Jul 03 14:14:38.577122 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.577089 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-whqcc"] Jul 03 14:14:38.577252 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.577178 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-whqcc" Jul 03 14:14:38.581522 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.581495 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 03 14:14:38.581720 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.581543 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-2744g\"" Jul 03 14:14:38.582020 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.581998 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 03 14:14:38.610471 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.610437 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/813d681d-addd-4164-a869-e5b1c63d4184-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-whqcc\" (UID: \"813d681d-addd-4164-a869-e5b1c63d4184\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-whqcc" Jul 03 14:14:38.610634 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.610492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhxs4\" (UniqueName: \"kubernetes.io/projected/813d681d-addd-4164-a869-e5b1c63d4184-kube-api-access-jhxs4\") pod \"cert-manager-operator-controller-manager-55c444bf86-whqcc\" (UID: \"813d681d-addd-4164-a869-e5b1c63d4184\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-whqcc" Jul 03 14:14:38.711115 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.711077 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/813d681d-addd-4164-a869-e5b1c63d4184-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-whqcc\" (UID: \"813d681d-addd-4164-a869-e5b1c63d4184\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-whqcc" Jul 03 14:14:38.711315 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.711132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhxs4\" (UniqueName: \"kubernetes.io/projected/813d681d-addd-4164-a869-e5b1c63d4184-kube-api-access-jhxs4\") pod \"cert-manager-operator-controller-manager-55c444bf86-whqcc\" (UID: \"813d681d-addd-4164-a869-e5b1c63d4184\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-whqcc" Jul 03 14:14:38.711514 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.711492 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/813d681d-addd-4164-a869-e5b1c63d4184-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-whqcc\" (UID: \"813d681d-addd-4164-a869-e5b1c63d4184\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-whqcc" Jul 03 14:14:38.721865 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.721825 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhxs4\" (UniqueName: \"kubernetes.io/projected/813d681d-addd-4164-a869-e5b1c63d4184-kube-api-access-jhxs4\") pod \"cert-manager-operator-controller-manager-55c444bf86-whqcc\" (UID: \"813d681d-addd-4164-a869-e5b1c63d4184\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-whqcc" Jul 03 14:14:38.888030 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:38.887939 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-whqcc" Jul 03 14:14:39.062609 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:39.062559 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-whqcc"] Jul 03 14:14:39.067054 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:14:39.067022 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813d681d_addd_4164_a869_e5b1c63d4184.slice/crio-5b3f10b525cffe24a872484325f1bbc1d151815904aa0d7cb90b1002ef255bfd WatchSource:0}: Error finding container 5b3f10b525cffe24a872484325f1bbc1d151815904aa0d7cb90b1002ef255bfd: Status 404 returned error can't find the container with id 5b3f10b525cffe24a872484325f1bbc1d151815904aa0d7cb90b1002ef255bfd Jul 03 14:14:39.695771 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:39.695729 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-whqcc" event={"ID":"813d681d-addd-4164-a869-e5b1c63d4184","Type":"ContainerStarted","Data":"5b3f10b525cffe24a872484325f1bbc1d151815904aa0d7cb90b1002ef255bfd"} Jul 03 14:14:41.704722 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:41.704684 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-whqcc" event={"ID":"813d681d-addd-4164-a869-e5b1c63d4184","Type":"ContainerStarted","Data":"7a41ae5e066dc787cea423775454e696d169b1507d8844d4ca6fae8decde9985"} Jul 03 14:14:41.729993 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:41.729934 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-whqcc" podStartSLOduration=1.6685814749999999 podStartE2EDuration="3.729919763s" totalImagesPullingTime="2.061338288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:14:41.727863344 +0000 UTC m=+360.084147975" watchObservedRunningTime="2026-07-03 14:14:41.729919763 +0000 UTC m=+360.086204383" Jul 03 14:14:49.414482 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:49.414437 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-drrcw"] Jul 03 14:14:49.417118 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:49.417089 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-drrcw" Jul 03 14:14:49.420592 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:49.420574 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mndjd\"" Jul 03 14:14:49.421844 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:49.421829 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 03 14:14:49.421908 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:49.421876 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 03 14:14:49.433254 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:49.433222 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-drrcw"] Jul 03 14:14:49.492760 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:49.492726 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxpck\" (UniqueName: \"kubernetes.io/projected/2bbf48fb-be87-4bc0-8902-8f1054e71a58-kube-api-access-pxpck\") pod \"cert-manager-cainjector-559cdd8c87-drrcw\" (UID: \"2bbf48fb-be87-4bc0-8902-8f1054e71a58\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-drrcw" Jul 03 14:14:49.492936 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:49.492772 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2bbf48fb-be87-4bc0-8902-8f1054e71a58-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-drrcw\" (UID: \"2bbf48fb-be87-4bc0-8902-8f1054e71a58\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-drrcw" Jul 03 14:14:49.593552 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:49.593518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxpck\" (UniqueName: \"kubernetes.io/projected/2bbf48fb-be87-4bc0-8902-8f1054e71a58-kube-api-access-pxpck\") pod \"cert-manager-cainjector-559cdd8c87-drrcw\" (UID: \"2bbf48fb-be87-4bc0-8902-8f1054e71a58\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-drrcw" Jul 03 14:14:49.593750 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:49.593568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2bbf48fb-be87-4bc0-8902-8f1054e71a58-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-drrcw\" (UID: \"2bbf48fb-be87-4bc0-8902-8f1054e71a58\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-drrcw" Jul 03 14:14:49.603456 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:49.603424 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2bbf48fb-be87-4bc0-8902-8f1054e71a58-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-drrcw\" (UID: \"2bbf48fb-be87-4bc0-8902-8f1054e71a58\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-drrcw" Jul 03 14:14:49.603570 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:49.603532 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxpck\" (UniqueName: \"kubernetes.io/projected/2bbf48fb-be87-4bc0-8902-8f1054e71a58-kube-api-access-pxpck\") pod \"cert-manager-cainjector-559cdd8c87-drrcw\" (UID: \"2bbf48fb-be87-4bc0-8902-8f1054e71a58\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-drrcw" Jul 03 14:14:49.726837 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:49.726751 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-drrcw" Jul 03 14:14:49.873157 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:49.873128 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-drrcw"] Jul 03 14:14:49.875591 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:14:49.875557 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bbf48fb_be87_4bc0_8902_8f1054e71a58.slice/crio-0b66b7ee43c7afbbe54c658c897e63c948598f52bc7ee971daf0b2ffd84e9f97 WatchSource:0}: Error finding container 0b66b7ee43c7afbbe54c658c897e63c948598f52bc7ee971daf0b2ffd84e9f97: Status 404 returned error can't find the container with id 0b66b7ee43c7afbbe54c658c897e63c948598f52bc7ee971daf0b2ffd84e9f97 Jul 03 14:14:50.746992 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:50.746951 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-drrcw" event={"ID":"2bbf48fb-be87-4bc0-8902-8f1054e71a58","Type":"ContainerStarted","Data":"0b66b7ee43c7afbbe54c658c897e63c948598f52bc7ee971daf0b2ffd84e9f97"} Jul 03 14:14:52.757169 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:52.757136 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-drrcw" event={"ID":"2bbf48fb-be87-4bc0-8902-8f1054e71a58","Type":"ContainerStarted","Data":"7557335c9d8cee59e7a0cda28e83e07dd8224228f102311f64c2cea4c76c988f"} Jul 03 14:14:52.774632 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:14:52.774585 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-drrcw" podStartSLOduration=1.485978227 podStartE2EDuration="3.774570257s" totalImagesPullingTime="2.28859203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:14:52.774156279 +0000 UTC m=+371.130440899" watchObservedRunningTime="2026-07-03 14:14:52.774570257 +0000 UTC m=+371.130854877" Jul 03 14:15:14.011393 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.011359 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6"] Jul 03 14:15:14.014208 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.014187 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6" Jul 03 14:15:14.019498 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.019478 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 03 14:15:14.019611 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.019565 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-d7mkw\"" Jul 03 14:15:14.020927 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.020907 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 03 14:15:14.028278 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.028248 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6"] Jul 03 14:15:14.099009 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.098968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw5gq\" (UniqueName: \"kubernetes.io/projected/e2e0501f-798d-4ee2-ab21-15293f8bdeb9-kube-api-access-mw5gq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6\" (UID: \"e2e0501f-798d-4ee2-ab21-15293f8bdeb9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6" Jul 03 14:15:14.099190 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.099022 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2e0501f-798d-4ee2-ab21-15293f8bdeb9-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6\" (UID: \"e2e0501f-798d-4ee2-ab21-15293f8bdeb9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6" Jul 03 14:15:14.099190 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.099044 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2e0501f-798d-4ee2-ab21-15293f8bdeb9-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6\" (UID: \"e2e0501f-798d-4ee2-ab21-15293f8bdeb9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6" Jul 03 14:15:14.200427 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.200395 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2e0501f-798d-4ee2-ab21-15293f8bdeb9-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6\" (UID: \"e2e0501f-798d-4ee2-ab21-15293f8bdeb9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6" Jul 03 14:15:14.200427 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.200430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2e0501f-798d-4ee2-ab21-15293f8bdeb9-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6\" (UID: \"e2e0501f-798d-4ee2-ab21-15293f8bdeb9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6" Jul 03 14:15:14.200720 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.200495 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw5gq\" (UniqueName: \"kubernetes.io/projected/e2e0501f-798d-4ee2-ab21-15293f8bdeb9-kube-api-access-mw5gq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6\" (UID: \"e2e0501f-798d-4ee2-ab21-15293f8bdeb9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6" Jul 03 14:15:14.200925 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.200902 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2e0501f-798d-4ee2-ab21-15293f8bdeb9-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6\" (UID: \"e2e0501f-798d-4ee2-ab21-15293f8bdeb9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6" Jul 03 14:15:14.200994 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.200947 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2e0501f-798d-4ee2-ab21-15293f8bdeb9-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6\" (UID: \"e2e0501f-798d-4ee2-ab21-15293f8bdeb9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6" Jul 03 14:15:14.215185 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.215160 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw5gq\" (UniqueName: \"kubernetes.io/projected/e2e0501f-798d-4ee2-ab21-15293f8bdeb9-kube-api-access-mw5gq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6\" (UID: \"e2e0501f-798d-4ee2-ab21-15293f8bdeb9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6" Jul 03 14:15:14.323455 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.323420 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6" Jul 03 14:15:14.455822 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.455798 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6"] Jul 03 14:15:14.458078 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:15:14.458052 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e0501f_798d_4ee2_ab21_15293f8bdeb9.slice/crio-72393e0c8910a0bbbaff1bf8b4191a853b68d2c92eb2b6ec54f04975cf89c6b4 WatchSource:0}: Error finding container 72393e0c8910a0bbbaff1bf8b4191a853b68d2c92eb2b6ec54f04975cf89c6b4: Status 404 returned error can't find the container with id 72393e0c8910a0bbbaff1bf8b4191a853b68d2c92eb2b6ec54f04975cf89c6b4 Jul 03 14:15:14.844391 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.844356 2576 generic.go:356] "Generic (PLEG): container finished" podID="e2e0501f-798d-4ee2-ab21-15293f8bdeb9" containerID="b5a3b343775f4746c1dd3f7e048ab3eb6b32a3fe7c1d7206b046103b2379d114" exitCode=0 Jul 03 14:15:14.844569 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.844422 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6" event={"ID":"e2e0501f-798d-4ee2-ab21-15293f8bdeb9","Type":"ContainerDied","Data":"b5a3b343775f4746c1dd3f7e048ab3eb6b32a3fe7c1d7206b046103b2379d114"} Jul 03 14:15:14.844569 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:14.844488 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6" event={"ID":"e2e0501f-798d-4ee2-ab21-15293f8bdeb9","Type":"ContainerStarted","Data":"72393e0c8910a0bbbaff1bf8b4191a853b68d2c92eb2b6ec54f04975cf89c6b4"} Jul 03 14:15:15.851489 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:15.851453 2576 generic.go:356] "Generic (PLEG): container finished" podID="e2e0501f-798d-4ee2-ab21-15293f8bdeb9" containerID="e8092756c88225cf44564cc9ff2bbee668e594a6ec887e3c998d27b4b67c9932" exitCode=0 Jul 03 14:15:15.851896 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:15.851550 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6" event={"ID":"e2e0501f-798d-4ee2-ab21-15293f8bdeb9","Type":"ContainerDied","Data":"e8092756c88225cf44564cc9ff2bbee668e594a6ec887e3c998d27b4b67c9932"} Jul 03 14:15:16.858362 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:16.858326 2576 generic.go:356] "Generic (PLEG): container finished" podID="e2e0501f-798d-4ee2-ab21-15293f8bdeb9" containerID="45bb1b919e7cc529bbd2ce39a1e67dc19acc73673015d4b2cbb43408d906c124" exitCode=0 Jul 03 14:15:16.858771 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:16.858368 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6" event={"ID":"e2e0501f-798d-4ee2-ab21-15293f8bdeb9","Type":"ContainerDied","Data":"45bb1b919e7cc529bbd2ce39a1e67dc19acc73673015d4b2cbb43408d906c124"} Jul 03 14:15:17.983242 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:17.983215 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6" Jul 03 14:15:18.134707 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.134588 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2e0501f-798d-4ee2-ab21-15293f8bdeb9-util\") pod \"e2e0501f-798d-4ee2-ab21-15293f8bdeb9\" (UID: \"e2e0501f-798d-4ee2-ab21-15293f8bdeb9\") " Jul 03 14:15:18.134883 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.134719 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw5gq\" (UniqueName: \"kubernetes.io/projected/e2e0501f-798d-4ee2-ab21-15293f8bdeb9-kube-api-access-mw5gq\") pod \"e2e0501f-798d-4ee2-ab21-15293f8bdeb9\" (UID: \"e2e0501f-798d-4ee2-ab21-15293f8bdeb9\") " Jul 03 14:15:18.134883 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.134768 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2e0501f-798d-4ee2-ab21-15293f8bdeb9-bundle\") pod \"e2e0501f-798d-4ee2-ab21-15293f8bdeb9\" (UID: \"e2e0501f-798d-4ee2-ab21-15293f8bdeb9\") " Jul 03 14:15:18.135538 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.135506 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e0501f-798d-4ee2-ab21-15293f8bdeb9-bundle" (OuterVolumeSpecName: "bundle") pod "e2e0501f-798d-4ee2-ab21-15293f8bdeb9" (UID: "e2e0501f-798d-4ee2-ab21-15293f8bdeb9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:15:18.137081 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.137058 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e0501f-798d-4ee2-ab21-15293f8bdeb9-kube-api-access-mw5gq" (OuterVolumeSpecName: "kube-api-access-mw5gq") pod "e2e0501f-798d-4ee2-ab21-15293f8bdeb9" (UID: "e2e0501f-798d-4ee2-ab21-15293f8bdeb9"). InnerVolumeSpecName "kube-api-access-mw5gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:15:18.140258 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.140220 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e0501f-798d-4ee2-ab21-15293f8bdeb9-util" (OuterVolumeSpecName: "util") pod "e2e0501f-798d-4ee2-ab21-15293f8bdeb9" (UID: "e2e0501f-798d-4ee2-ab21-15293f8bdeb9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:15:18.235719 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.235682 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw5gq\" (UniqueName: \"kubernetes.io/projected/e2e0501f-798d-4ee2-ab21-15293f8bdeb9-kube-api-access-mw5gq\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:15:18.235719 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.235712 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2e0501f-798d-4ee2-ab21-15293f8bdeb9-bundle\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:15:18.235719 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.235721 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2e0501f-798d-4ee2-ab21-15293f8bdeb9-util\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:15:18.658369 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.658333 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn"] Jul 03 14:15:18.658617 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.658604 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2e0501f-798d-4ee2-ab21-15293f8bdeb9" containerName="extract" Jul 03 14:15:18.658688 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.658618 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2e0501f-798d-4ee2-ab21-15293f8bdeb9" containerName="extract" Jul 03 14:15:18.658688 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.658635 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2e0501f-798d-4ee2-ab21-15293f8bdeb9" containerName="pull" Jul 03 14:15:18.658688 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.658640 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2e0501f-798d-4ee2-ab21-15293f8bdeb9" containerName="pull" Jul 03 14:15:18.658688 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.658665 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2e0501f-798d-4ee2-ab21-15293f8bdeb9" containerName="util" Jul 03 14:15:18.658688 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.658671 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2e0501f-798d-4ee2-ab21-15293f8bdeb9" containerName="util" Jul 03 14:15:18.658841 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.658714 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2e0501f-798d-4ee2-ab21-15293f8bdeb9" containerName="extract" Jul 03 14:15:18.660873 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.660858 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn" Jul 03 14:15:18.664521 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.664487 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 03 14:15:18.665987 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.665940 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 03 14:15:18.666099 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.665997 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 03 14:15:18.666099 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.666021 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-w2zrb\"" Jul 03 14:15:18.666193 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.665946 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 03 14:15:18.666262 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.666244 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 03 14:15:18.678107 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.678084 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn"] Jul 03 14:15:18.839915 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.839881 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7433f219-4c0d-45d2-b727-1ade4503e8b0-cert\") pod \"lws-controller-manager-78cbbf6689-fh8dn\" (UID: \"7433f219-4c0d-45d2-b727-1ade4503e8b0\") " pod="openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn" Jul 03 14:15:18.839915 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.839922 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6g7d\" (UniqueName: \"kubernetes.io/projected/7433f219-4c0d-45d2-b727-1ade4503e8b0-kube-api-access-c6g7d\") pod \"lws-controller-manager-78cbbf6689-fh8dn\" (UID: \"7433f219-4c0d-45d2-b727-1ade4503e8b0\") " pod="openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn" Jul 03 14:15:18.840137 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.839955 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7433f219-4c0d-45d2-b727-1ade4503e8b0-manager-config\") pod \"lws-controller-manager-78cbbf6689-fh8dn\" (UID: \"7433f219-4c0d-45d2-b727-1ade4503e8b0\") " pod="openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn" Jul 03 14:15:18.840137 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.840037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7433f219-4c0d-45d2-b727-1ade4503e8b0-metrics-cert\") pod \"lws-controller-manager-78cbbf6689-fh8dn\" (UID: \"7433f219-4c0d-45d2-b727-1ade4503e8b0\") " pod="openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn" Jul 03 14:15:18.876068 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.876029 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6" event={"ID":"e2e0501f-798d-4ee2-ab21-15293f8bdeb9","Type":"ContainerDied","Data":"72393e0c8910a0bbbaff1bf8b4191a853b68d2c92eb2b6ec54f04975cf89c6b4"} Jul 03 14:15:18.876068 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.876066 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7dkg6" Jul 03 14:15:18.876263 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.876113 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72393e0c8910a0bbbaff1bf8b4191a853b68d2c92eb2b6ec54f04975cf89c6b4" Jul 03 14:15:18.941004 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.940899 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7433f219-4c0d-45d2-b727-1ade4503e8b0-metrics-cert\") pod \"lws-controller-manager-78cbbf6689-fh8dn\" (UID: \"7433f219-4c0d-45d2-b727-1ade4503e8b0\") " pod="openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn" Jul 03 14:15:18.941004 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.940962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7433f219-4c0d-45d2-b727-1ade4503e8b0-cert\") pod \"lws-controller-manager-78cbbf6689-fh8dn\" (UID: \"7433f219-4c0d-45d2-b727-1ade4503e8b0\") " pod="openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn" Jul 03 14:15:18.941004 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.940996 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6g7d\" (UniqueName: \"kubernetes.io/projected/7433f219-4c0d-45d2-b727-1ade4503e8b0-kube-api-access-c6g7d\") pod \"lws-controller-manager-78cbbf6689-fh8dn\" (UID: \"7433f219-4c0d-45d2-b727-1ade4503e8b0\") " pod="openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn" Jul 03 14:15:18.941219 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.941054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7433f219-4c0d-45d2-b727-1ade4503e8b0-manager-config\") pod \"lws-controller-manager-78cbbf6689-fh8dn\" (UID: \"7433f219-4c0d-45d2-b727-1ade4503e8b0\") " pod="openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn" Jul 03 14:15:18.941727 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.941709 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7433f219-4c0d-45d2-b727-1ade4503e8b0-manager-config\") pod \"lws-controller-manager-78cbbf6689-fh8dn\" (UID: \"7433f219-4c0d-45d2-b727-1ade4503e8b0\") " pod="openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn" Jul 03 14:15:18.943553 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.943523 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7433f219-4c0d-45d2-b727-1ade4503e8b0-metrics-cert\") pod \"lws-controller-manager-78cbbf6689-fh8dn\" (UID: \"7433f219-4c0d-45d2-b727-1ade4503e8b0\") " pod="openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn" Jul 03 14:15:18.943711 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.943688 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7433f219-4c0d-45d2-b727-1ade4503e8b0-cert\") pod \"lws-controller-manager-78cbbf6689-fh8dn\" (UID: \"7433f219-4c0d-45d2-b727-1ade4503e8b0\") " pod="openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn" Jul 03 14:15:18.951542 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.951515 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6g7d\" (UniqueName: \"kubernetes.io/projected/7433f219-4c0d-45d2-b727-1ade4503e8b0-kube-api-access-c6g7d\") pod \"lws-controller-manager-78cbbf6689-fh8dn\" (UID: \"7433f219-4c0d-45d2-b727-1ade4503e8b0\") " pod="openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn" Jul 03 14:15:18.970426 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:18.970384 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn" Jul 03 14:15:19.115959 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:19.115923 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn"] Jul 03 14:15:19.119302 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:15:19.119275 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7433f219_4c0d_45d2_b727_1ade4503e8b0.slice/crio-1f27c2b880591fb766a907c261ef0c040667af62190200fc524c71d5f64e295c WatchSource:0}: Error finding container 1f27c2b880591fb766a907c261ef0c040667af62190200fc524c71d5f64e295c: Status 404 returned error can't find the container with id 1f27c2b880591fb766a907c261ef0c040667af62190200fc524c71d5f64e295c Jul 03 14:15:19.881198 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:19.881163 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn" event={"ID":"7433f219-4c0d-45d2-b727-1ade4503e8b0","Type":"ContainerStarted","Data":"1f27c2b880591fb766a907c261ef0c040667af62190200fc524c71d5f64e295c"} Jul 03 14:15:21.891957 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:21.891921 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn" event={"ID":"7433f219-4c0d-45d2-b727-1ade4503e8b0","Type":"ContainerStarted","Data":"18975df168da96ae82c6a9d92d7ad309cafae6a48dc143599e5c58d87e83dcf9"} Jul 03 14:15:21.892385 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:21.892025 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn" Jul 03 14:15:21.910794 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:21.910747 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn" podStartSLOduration=1.297069842 podStartE2EDuration="3.910731399s" totalImagesPullingTime="2.613661557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:15:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:15:21.909750423 +0000 UTC m=+400.266035045" watchObservedRunningTime="2026-07-03 14:15:21.910731399 +0000 UTC m=+400.267016020" Jul 03 14:15:28.329084 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.329049 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c"] Jul 03 14:15:28.331682 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.331645 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" Jul 03 14:15:28.335280 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.335257 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 03 14:15:28.335405 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.335297 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 03 14:15:28.335405 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.335373 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xwgf7\"" Jul 03 14:15:28.335823 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.335791 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 03 14:15:28.339672 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.339635 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 03 14:15:28.369375 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.369338 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c"] Jul 03 14:15:28.422183 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.422147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/11ed09d0-3eca-431e-8f56-62433bc47804-apiservice-cert\") pod \"opendatahub-operator-controller-manager-977f7bbf6-7cn9c\" (UID: \"11ed09d0-3eca-431e-8f56-62433bc47804\") " pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" Jul 03 14:15:28.422183 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.422186 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzgr4\" (UniqueName: \"kubernetes.io/projected/11ed09d0-3eca-431e-8f56-62433bc47804-kube-api-access-pzgr4\") pod \"opendatahub-operator-controller-manager-977f7bbf6-7cn9c\" (UID: \"11ed09d0-3eca-431e-8f56-62433bc47804\") " pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" Jul 03 14:15:28.422441 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.422216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11ed09d0-3eca-431e-8f56-62433bc47804-manifests\") pod \"opendatahub-operator-controller-manager-977f7bbf6-7cn9c\" (UID: \"11ed09d0-3eca-431e-8f56-62433bc47804\") " pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" Jul 03 14:15:28.422441 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.422311 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11ed09d0-3eca-431e-8f56-62433bc47804-webhook-cert\") pod \"opendatahub-operator-controller-manager-977f7bbf6-7cn9c\" (UID: \"11ed09d0-3eca-431e-8f56-62433bc47804\") " pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" Jul 03 14:15:28.422441 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.422356 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11ed09d0-3eca-431e-8f56-62433bc47804-tmp\") pod \"opendatahub-operator-controller-manager-977f7bbf6-7cn9c\" (UID: \"11ed09d0-3eca-431e-8f56-62433bc47804\") " pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" Jul 03 14:15:28.523484 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.523450 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11ed09d0-3eca-431e-8f56-62433bc47804-manifests\") pod \"opendatahub-operator-controller-manager-977f7bbf6-7cn9c\" (UID: \"11ed09d0-3eca-431e-8f56-62433bc47804\") " pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" Jul 03 14:15:28.523648 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.523501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11ed09d0-3eca-431e-8f56-62433bc47804-webhook-cert\") pod \"opendatahub-operator-controller-manager-977f7bbf6-7cn9c\" (UID: \"11ed09d0-3eca-431e-8f56-62433bc47804\") " pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" Jul 03 14:15:28.523648 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.523619 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11ed09d0-3eca-431e-8f56-62433bc47804-tmp\") pod \"opendatahub-operator-controller-manager-977f7bbf6-7cn9c\" (UID: \"11ed09d0-3eca-431e-8f56-62433bc47804\") " pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" Jul 03 14:15:28.523847 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.523702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/11ed09d0-3eca-431e-8f56-62433bc47804-apiservice-cert\") pod \"opendatahub-operator-controller-manager-977f7bbf6-7cn9c\" (UID: \"11ed09d0-3eca-431e-8f56-62433bc47804\") " pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" Jul 03 14:15:28.523847 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.523741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzgr4\" (UniqueName: \"kubernetes.io/projected/11ed09d0-3eca-431e-8f56-62433bc47804-kube-api-access-pzgr4\") pod \"opendatahub-operator-controller-manager-977f7bbf6-7cn9c\" (UID: \"11ed09d0-3eca-431e-8f56-62433bc47804\") " pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" Jul 03 14:15:28.524051 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.524026 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11ed09d0-3eca-431e-8f56-62433bc47804-manifests\") pod \"opendatahub-operator-controller-manager-977f7bbf6-7cn9c\" (UID: \"11ed09d0-3eca-431e-8f56-62433bc47804\") " pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" Jul 03 14:15:28.526157 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.526131 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11ed09d0-3eca-431e-8f56-62433bc47804-webhook-cert\") pod \"opendatahub-operator-controller-manager-977f7bbf6-7cn9c\" (UID: \"11ed09d0-3eca-431e-8f56-62433bc47804\") " pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" Jul 03 14:15:28.526233 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.526133 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11ed09d0-3eca-431e-8f56-62433bc47804-tmp\") pod \"opendatahub-operator-controller-manager-977f7bbf6-7cn9c\" (UID: \"11ed09d0-3eca-431e-8f56-62433bc47804\") " pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" Jul 03 14:15:28.526233 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.526137 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/11ed09d0-3eca-431e-8f56-62433bc47804-apiservice-cert\") pod \"opendatahub-operator-controller-manager-977f7bbf6-7cn9c\" (UID: \"11ed09d0-3eca-431e-8f56-62433bc47804\") " pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" Jul 03 14:15:28.537200 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.537175 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzgr4\" (UniqueName: \"kubernetes.io/projected/11ed09d0-3eca-431e-8f56-62433bc47804-kube-api-access-pzgr4\") pod \"opendatahub-operator-controller-manager-977f7bbf6-7cn9c\" (UID: \"11ed09d0-3eca-431e-8f56-62433bc47804\") " pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" Jul 03 14:15:28.642612 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.642527 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" Jul 03 14:15:28.794244 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.794210 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c"] Jul 03 14:15:28.798560 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:15:28.798525 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ed09d0_3eca_431e_8f56_62433bc47804.slice/crio-bfdc6ac6ba94766fec2e660fa02713e914ff58f971b18f89dc54006702056063 WatchSource:0}: Error finding container bfdc6ac6ba94766fec2e660fa02713e914ff58f971b18f89dc54006702056063: Status 404 returned error can't find the container with id bfdc6ac6ba94766fec2e660fa02713e914ff58f971b18f89dc54006702056063 Jul 03 14:15:28.923420 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:28.923326 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" event={"ID":"11ed09d0-3eca-431e-8f56-62433bc47804","Type":"ContainerStarted","Data":"bfdc6ac6ba94766fec2e660fa02713e914ff58f971b18f89dc54006702056063"} Jul 03 14:15:31.939198 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:31.939163 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" event={"ID":"11ed09d0-3eca-431e-8f56-62433bc47804","Type":"ContainerStarted","Data":"f58cbd47af1209e0eebf588b50c35d0306925b1d21a39e27be8c01ccae14bd4f"} Jul 03 14:15:32.898869 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:32.898840 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-78cbbf6689-fh8dn" Jul 03 14:15:32.945595 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:32.945560 2576 generic.go:356] "Generic (PLEG): container finished" podID="11ed09d0-3eca-431e-8f56-62433bc47804" containerID="f58cbd47af1209e0eebf588b50c35d0306925b1d21a39e27be8c01ccae14bd4f" exitCode=0 Jul 03 14:15:32.946033 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:32.945642 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" event={"ID":"11ed09d0-3eca-431e-8f56-62433bc47804","Type":"ContainerDied","Data":"f58cbd47af1209e0eebf588b50c35d0306925b1d21a39e27be8c01ccae14bd4f"} Jul 03 14:15:33.952425 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:33.952383 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" event={"ID":"11ed09d0-3eca-431e-8f56-62433bc47804","Type":"ContainerStarted","Data":"663fcc73749a4487742948804866fba54ecc7d4ccb324a0f2538b5c370b4ffcb"} Jul 03 14:15:33.952847 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:33.952526 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" Jul 03 14:15:33.980374 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:33.980324 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" podStartSLOduration=1.7055548470000002 podStartE2EDuration="5.980307736s" totalImagesPullingTime="3.24660847s" totalInitContainerRuntime="1.028144419s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:15:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:15:33.979851836 +0000 UTC m=+412.336136456" watchObservedRunningTime="2026-07-03 14:15:33.980307736 +0000 UTC m=+412.336592355" Jul 03 14:15:44.959309 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:44.959277 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-977f7bbf6-7cn9c" Jul 03 14:15:55.635862 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.635833 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq"] Jul 03 14:15:55.644182 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.644160 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.647257 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.647232 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 03 14:15:55.647378 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.647267 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-8ftfm\"" Jul 03 14:15:55.652902 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.652867 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq"] Jul 03 14:15:55.756364 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.756330 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.756546 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.756386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwbvm\" (UniqueName: \"kubernetes.io/projected/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-kube-api-access-kwbvm\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.756546 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.756442 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.756546 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.756466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.756546 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.756527 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.756709 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.756572 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.756709 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.756591 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.756709 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.756612 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.756709 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.756650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.857726 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.857690 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.857896 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.857744 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwbvm\" (UniqueName: \"kubernetes.io/projected/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-kube-api-access-kwbvm\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.857896 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.857769 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.857896 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.857788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.857896 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.857804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.858069 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.857940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.858069 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.857981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.858069 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.858018 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.858069 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.858061 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.858267 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.858223 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.858323 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.858268 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.858445 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.858408 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.858561 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.858543 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.858686 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.858642 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.860261 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.860237 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.860438 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.860415 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.871447 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.871417 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.881748 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.881726 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwbvm\" (UniqueName: \"kubernetes.io/projected/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-kube-api-access-kwbvm\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:55.954808 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:55.954723 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:15:56.016440 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.016304 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn"] Jul 03 14:15:56.019309 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.019282 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.035832 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.035799 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn"] Jul 03 14:15:56.114556 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.114517 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp"] Jul 03 14:15:56.117007 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.116991 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.122939 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.122908 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 03 14:15:56.123411 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.123387 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 03 14:15:56.123629 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.123609 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 03 14:15:56.129258 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.129238 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tphwm\"" Jul 03 14:15:56.141800 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.140921 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq"] Jul 03 14:15:56.143394 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.143309 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp"] Jul 03 14:15:56.143807 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:15:56.143779 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7f89c4_a34d_4f0d_bad4_e77bef6c9da8.slice/crio-2aaebd97b66b91fa8a8468ca2a8bda0a6daf69d1bb89278200f9a91f8e906f2f WatchSource:0}: Error finding container 2aaebd97b66b91fa8a8468ca2a8bda0a6daf69d1bb89278200f9a91f8e906f2f: Status 404 returned error can't find the container with id 2aaebd97b66b91fa8a8468ca2a8bda0a6daf69d1bb89278200f9a91f8e906f2f Jul 03 14:15:56.161502 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.161474 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85ll6\" (UniqueName: \"kubernetes.io/projected/c01fd94e-5c98-4c4e-a984-9e21f891425b-kube-api-access-85ll6\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.161899 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.161876 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01fd94e-5c98-4c4e-a984-9e21f891425b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.162586 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.162494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01fd94e-5c98-4c4e-a984-9e21f891425b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.162586 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.162529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01fd94e-5c98-4c4e-a984-9e21f891425b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.162586 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.162562 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01fd94e-5c98-4c4e-a984-9e21f891425b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.162827 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.162614 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01fd94e-5c98-4c4e-a984-9e21f891425b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.162827 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.162678 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01fd94e-5c98-4c4e-a984-9e21f891425b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.162827 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.162704 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01fd94e-5c98-4c4e-a984-9e21f891425b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.162827 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.162733 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01fd94e-5c98-4c4e-a984-9e21f891425b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.263470 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.263368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2xdq\" (UniqueName: \"kubernetes.io/projected/c49627e5-6591-4449-9ffd-7016ac083996-kube-api-access-f2xdq\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.263470 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.263429 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01fd94e-5c98-4c4e-a984-9e21f891425b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.263707 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.263479 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01fd94e-5c98-4c4e-a984-9e21f891425b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.263707 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.263516 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c49627e5-6591-4449-9ffd-7016ac083996-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.263707 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.263541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01fd94e-5c98-4c4e-a984-9e21f891425b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.263707 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.263581 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49627e5-6591-4449-9ffd-7016ac083996-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.263937 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.263711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85ll6\" (UniqueName: \"kubernetes.io/projected/c01fd94e-5c98-4c4e-a984-9e21f891425b-kube-api-access-85ll6\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.263937 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.263750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01fd94e-5c98-4c4e-a984-9e21f891425b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.263937 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.263787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01fd94e-5c98-4c4e-a984-9e21f891425b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.263937 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.263810 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01fd94e-5c98-4c4e-a984-9e21f891425b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.263937 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.263827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49627e5-6591-4449-9ffd-7016ac083996-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.263937 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.263836 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01fd94e-5c98-4c4e-a984-9e21f891425b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.263937 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.263849 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01fd94e-5c98-4c4e-a984-9e21f891425b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.264245 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.263927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c49627e5-6591-4449-9ffd-7016ac083996-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.264245 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.263973 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c49627e5-6591-4449-9ffd-7016ac083996-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.264245 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.263996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c49627e5-6591-4449-9ffd-7016ac083996-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.264245 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.264086 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01fd94e-5c98-4c4e-a984-9e21f891425b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.264245 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.264119 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01fd94e-5c98-4c4e-a984-9e21f891425b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.264245 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.264145 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01fd94e-5c98-4c4e-a984-9e21f891425b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.264571 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.264554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01fd94e-5c98-4c4e-a984-9e21f891425b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.264830 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.264808 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01fd94e-5c98-4c4e-a984-9e21f891425b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.266019 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.265998 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01fd94e-5c98-4c4e-a984-9e21f891425b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.266795 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.266777 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01fd94e-5c98-4c4e-a984-9e21f891425b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.274411 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.274382 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01fd94e-5c98-4c4e-a984-9e21f891425b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.275230 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.275203 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85ll6\" (UniqueName: \"kubernetes.io/projected/c01fd94e-5c98-4c4e-a984-9e21f891425b-kube-api-access-85ll6\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x8sgn\" (UID: \"c01fd94e-5c98-4c4e-a984-9e21f891425b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.335348 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.335314 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:15:56.365345 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.365313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2xdq\" (UniqueName: \"kubernetes.io/projected/c49627e5-6591-4449-9ffd-7016ac083996-kube-api-access-f2xdq\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.365484 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.365364 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c49627e5-6591-4449-9ffd-7016ac083996-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.365484 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.365400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49627e5-6591-4449-9ffd-7016ac083996-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.365484 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.365479 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49627e5-6591-4449-9ffd-7016ac083996-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.365644 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.365511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c49627e5-6591-4449-9ffd-7016ac083996-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.365644 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.365533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c49627e5-6591-4449-9ffd-7016ac083996-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.365644 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.365557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c49627e5-6591-4449-9ffd-7016ac083996-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.366563 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.366535 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c49627e5-6591-4449-9ffd-7016ac083996-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.367886 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.367858 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c49627e5-6591-4449-9ffd-7016ac083996-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.368045 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.368022 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c49627e5-6591-4449-9ffd-7016ac083996-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.368300 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.368284 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49627e5-6591-4449-9ffd-7016ac083996-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.368342 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.368302 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c49627e5-6591-4449-9ffd-7016ac083996-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.382321 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.382286 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49627e5-6591-4449-9ffd-7016ac083996-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.382530 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.382515 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2xdq\" (UniqueName: \"kubernetes.io/projected/c49627e5-6591-4449-9ffd-7016ac083996-kube-api-access-f2xdq\") pod \"istiod-openshift-gateway-94bb8fbfd-q6mtp\" (UID: \"c49627e5-6591-4449-9ffd-7016ac083996\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.428051 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.428015 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:15:56.498195 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.498165 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn"] Jul 03 14:15:56.500865 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:15:56.500838 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01fd94e_5c98_4c4e_a984_9e21f891425b.slice/crio-4c9819f3fa105bc5fd20faa1bcbe388192875a8bc4d1eca97dc8d8574eec088a WatchSource:0}: Error finding container 4c9819f3fa105bc5fd20faa1bcbe388192875a8bc4d1eca97dc8d8574eec088a: Status 404 returned error can't find the container with id 4c9819f3fa105bc5fd20faa1bcbe388192875a8bc4d1eca97dc8d8574eec088a Jul 03 14:15:56.652722 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:56.652675 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp"] Jul 03 14:15:56.654788 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:15:56.654761 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49627e5_6591_4449_9ffd_7016ac083996.slice/crio-3c5c5939c8227ea34a8907288a0a24355f8fad2d5041a44dc6de4742c60fb5ac WatchSource:0}: Error finding container 3c5c5939c8227ea34a8907288a0a24355f8fad2d5041a44dc6de4742c60fb5ac: Status 404 returned error can't find the container with id 3c5c5939c8227ea34a8907288a0a24355f8fad2d5041a44dc6de4742c60fb5ac Jul 03 14:15:57.066140 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:57.066092 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" event={"ID":"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8","Type":"ContainerStarted","Data":"2aaebd97b66b91fa8a8468ca2a8bda0a6daf69d1bb89278200f9a91f8e906f2f"} Jul 03 14:15:57.068834 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:57.068799 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" event={"ID":"c49627e5-6591-4449-9ffd-7016ac083996","Type":"ContainerStarted","Data":"3c5c5939c8227ea34a8907288a0a24355f8fad2d5041a44dc6de4742c60fb5ac"} Jul 03 14:15:57.071746 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:15:57.071680 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" event={"ID":"c01fd94e-5c98-4c4e-a984-9e21f891425b","Type":"ContainerStarted","Data":"4c9819f3fa105bc5fd20faa1bcbe388192875a8bc4d1eca97dc8d8574eec088a"} Jul 03 14:16:00.567931 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:00.567889 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:16:00.568328 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:00.567967 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:16:00.568328 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:00.568004 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:16:00.568328 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:00.567955 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:16:00.568328 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:00.568113 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:16:00.575369 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:00.575343 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:16:00.575445 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:00.575412 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:16:00.575508 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:00.575458 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:16:01.096193 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:01.096157 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" event={"ID":"c49627e5-6591-4449-9ffd-7016ac083996","Type":"ContainerStarted","Data":"ee45a6984707f198e70dafbc36c5b02128f6540b0b54543acdd41a8274aff1dd"} Jul 03 14:16:01.096420 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:01.096350 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:16:01.098187 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:01.098153 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" event={"ID":"c01fd94e-5c98-4c4e-a984-9e21f891425b","Type":"ContainerStarted","Data":"7c7cd2322aad6763e6ec1b04dac7a64ed3410eda1d9e1bba21c54a9e0f212651"} Jul 03 14:16:01.098543 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:01.098514 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-q6mtp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 03 14:16:01.098688 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:01.098577 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" podUID="c49627e5-6591-4449-9ffd-7016ac083996" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 03 14:16:01.099778 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:01.099753 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" event={"ID":"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8","Type":"ContainerStarted","Data":"b69e1653c4331142b1c27e52840f19244c6d9f40193274cf8b7e231ab6d3475f"} Jul 03 14:16:01.203068 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:01.203016 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" podStartSLOduration=1.292082421 podStartE2EDuration="5.202998956s" totalImagesPullingTime="3.910916535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:16:01.201149403 +0000 UTC m=+439.557434025" watchObservedRunningTime="2026-07-03 14:16:01.202998956 +0000 UTC m=+439.559283553" Jul 03 14:16:01.246285 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:01.246219 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" podStartSLOduration=2.181447734 podStartE2EDuration="6.246198024s" totalImagesPullingTime="4.06475029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:16:01.244072337 +0000 UTC m=+439.600356970" watchObservedRunningTime="2026-07-03 14:16:01.246198024 +0000 UTC m=+439.602482646" Jul 03 14:16:01.290324 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:01.290273 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" podStartSLOduration=1.861328111 podStartE2EDuration="6.290255232s" totalImagesPullingTime="4.428927121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:16:01.289516875 +0000 UTC m=+439.645801496" watchObservedRunningTime="2026-07-03 14:16:01.290255232 +0000 UTC m=+439.646539851" Jul 03 14:16:01.335884 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:01.335840 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:16:01.347942 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:01.347849 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:16:01.955767 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:01.955680 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:16:01.957115 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:01.957090 2576 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Jul 03 14:16:01.957225 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:01.957143 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" podUID="bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Jul 03 14:16:02.104955 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:02.104915 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:16:02.105745 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:02.105726 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-q6mtp" Jul 03 14:16:02.106066 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:02.106053 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x8sgn" Jul 03 14:16:02.243700 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:02.243597 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq"] Jul 03 14:16:02.955463 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:02.955425 2576 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Jul 03 14:16:02.955632 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:02.955485 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" podUID="bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Jul 03 14:16:03.955866 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:03.955834 2576 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Jul 03 14:16:03.956245 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:03.955889 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" podUID="bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Jul 03 14:16:04.113306 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:04.113266 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" podUID="bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8" containerName="istio-proxy" containerID="cri-o://b69e1653c4331142b1c27e52840f19244c6d9f40193274cf8b7e231ab6d3475f" gracePeriod=30 Jul 03 14:16:09.381039 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.380974 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:16:09.483471 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.483440 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-token\") pod \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " Jul 03 14:16:09.483471 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.483485 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwbvm\" (UniqueName: \"kubernetes.io/projected/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-kube-api-access-kwbvm\") pod \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " Jul 03 14:16:09.483763 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.483519 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-envoy\") pod \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " Jul 03 14:16:09.483763 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.483553 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-workload-certs\") pod \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " Jul 03 14:16:09.483763 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.483710 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-data\") pod \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " Jul 03 14:16:09.483903 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.483766 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-credential-socket\") pod \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " Jul 03 14:16:09.483903 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.483795 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-workload-socket\") pod \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " Jul 03 14:16:09.483903 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.483836 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-podinfo\") pod \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " Jul 03 14:16:09.483903 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.483842 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8" (UID: "bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:16:09.483903 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.483863 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istiod-ca-cert\") pod \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\" (UID: \"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8\") " Jul 03 14:16:09.484126 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.483957 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-data" (OuterVolumeSpecName: "istio-data") pod "bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8" (UID: "bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:16:09.484126 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.484008 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8" (UID: "bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:16:09.484232 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.484156 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8" (UID: "bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:16:09.484232 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.484177 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-workload-certs\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:16:09.484232 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.484205 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-data\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:16:09.484383 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.484239 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-workload-socket\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:16:09.484602 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.484566 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8" (UID: "bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:16:09.486046 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.486020 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8" (UID: "bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:16:09.486118 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.486046 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-token" (OuterVolumeSpecName: "istio-token") pod "bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8" (UID: "bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:16:09.486157 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.486125 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-kube-api-access-kwbvm" (OuterVolumeSpecName: "kube-api-access-kwbvm") pod "bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8" (UID: "bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8"). InnerVolumeSpecName "kube-api-access-kwbvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:16:09.486222 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.486207 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8" (UID: "bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 03 14:16:09.584642 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.584611 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-credential-socket\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:16:09.584642 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.584642 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-podinfo\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:16:09.584642 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.584676 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istiod-ca-cert\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:16:09.584890 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.584684 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-token\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:16:09.584890 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.584698 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwbvm\" (UniqueName: \"kubernetes.io/projected/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-kube-api-access-kwbvm\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:16:09.584890 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:09.584708 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8-istio-envoy\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:16:10.141682 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:10.141620 2576 generic.go:356] "Generic (PLEG): container finished" podID="bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8" containerID="b69e1653c4331142b1c27e52840f19244c6d9f40193274cf8b7e231ab6d3475f" exitCode=0 Jul 03 14:16:10.141682 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:10.141680 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" Jul 03 14:16:10.141903 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:10.141693 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" event={"ID":"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8","Type":"ContainerDied","Data":"b69e1653c4331142b1c27e52840f19244c6d9f40193274cf8b7e231ab6d3475f"} Jul 03 14:16:10.141903 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:10.141751 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq" event={"ID":"bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8","Type":"ContainerDied","Data":"2aaebd97b66b91fa8a8468ca2a8bda0a6daf69d1bb89278200f9a91f8e906f2f"} Jul 03 14:16:10.141903 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:10.141778 2576 scope.go:117] "RemoveContainer" containerID="b69e1653c4331142b1c27e52840f19244c6d9f40193274cf8b7e231ab6d3475f" Jul 03 14:16:10.152576 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:10.152556 2576 scope.go:117] "RemoveContainer" containerID="b69e1653c4331142b1c27e52840f19244c6d9f40193274cf8b7e231ab6d3475f" Jul 03 14:16:10.152939 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:16:10.152916 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b69e1653c4331142b1c27e52840f19244c6d9f40193274cf8b7e231ab6d3475f\": container with ID starting with b69e1653c4331142b1c27e52840f19244c6d9f40193274cf8b7e231ab6d3475f not found: ID does not exist" containerID="b69e1653c4331142b1c27e52840f19244c6d9f40193274cf8b7e231ab6d3475f" Jul 03 14:16:10.153001 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:10.152949 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b69e1653c4331142b1c27e52840f19244c6d9f40193274cf8b7e231ab6d3475f"} err="failed to get container status \"b69e1653c4331142b1c27e52840f19244c6d9f40193274cf8b7e231ab6d3475f\": rpc error: code = NotFound desc = could not find container \"b69e1653c4331142b1c27e52840f19244c6d9f40193274cf8b7e231ab6d3475f\": container with ID starting with b69e1653c4331142b1c27e52840f19244c6d9f40193274cf8b7e231ab6d3475f not found: ID does not exist" Jul 03 14:16:10.174244 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:10.174208 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq"] Jul 03 14:16:10.174635 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:10.174619 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-5d8bq"] Jul 03 14:16:12.168790 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:16:12.168756 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8" path="/var/lib/kubelet/pods/bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8/volumes" Jul 03 14:18:42.069329 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:42.069300 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:18:42.070825 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:42.070798 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:18:42.072237 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:42.072217 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:18:42.073611 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:42.073595 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:18:51.225927 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:51.225894 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f6cdd9cc7-d2hzz"] Jul 03 14:18:51.226382 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:51.226190 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8" containerName="istio-proxy" Jul 03 14:18:51.226382 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:51.226201 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8" containerName="istio-proxy" Jul 03 14:18:51.226382 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:51.226254 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb7f89c4-a34d-4f0d-bad4-e77bef6c9da8" containerName="istio-proxy" Jul 03 14:18:51.229018 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:51.228998 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f6cdd9cc7-d2hzz" Jul 03 14:18:51.232081 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:51.232052 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 03 14:18:51.232201 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:51.232118 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dlc6d\"" Jul 03 14:18:51.233353 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:51.233338 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 03 14:18:51.246534 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:51.246502 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f6cdd9cc7-d2hzz"] Jul 03 14:18:51.350365 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:51.350331 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwf6h\" (UniqueName: \"kubernetes.io/projected/be9798ef-bbf7-43e1-b485-ae851b3cdb96-kube-api-access-mwf6h\") pod \"maas-controller-f6cdd9cc7-d2hzz\" (UID: \"be9798ef-bbf7-43e1-b485-ae851b3cdb96\") " pod="opendatahub/maas-controller-f6cdd9cc7-d2hzz" Jul 03 14:18:51.350533 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:51.350407 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be9798ef-bbf7-43e1-b485-ae851b3cdb96-webhook-cert\") pod \"maas-controller-f6cdd9cc7-d2hzz\" (UID: \"be9798ef-bbf7-43e1-b485-ae851b3cdb96\") " pod="opendatahub/maas-controller-f6cdd9cc7-d2hzz" Jul 03 14:18:51.451226 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:51.451196 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be9798ef-bbf7-43e1-b485-ae851b3cdb96-webhook-cert\") pod \"maas-controller-f6cdd9cc7-d2hzz\" (UID: \"be9798ef-bbf7-43e1-b485-ae851b3cdb96\") " pod="opendatahub/maas-controller-f6cdd9cc7-d2hzz" Jul 03 14:18:51.451394 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:51.451253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwf6h\" (UniqueName: \"kubernetes.io/projected/be9798ef-bbf7-43e1-b485-ae851b3cdb96-kube-api-access-mwf6h\") pod \"maas-controller-f6cdd9cc7-d2hzz\" (UID: \"be9798ef-bbf7-43e1-b485-ae851b3cdb96\") " pod="opendatahub/maas-controller-f6cdd9cc7-d2hzz" Jul 03 14:18:51.453869 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:51.453846 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be9798ef-bbf7-43e1-b485-ae851b3cdb96-webhook-cert\") pod \"maas-controller-f6cdd9cc7-d2hzz\" (UID: \"be9798ef-bbf7-43e1-b485-ae851b3cdb96\") " pod="opendatahub/maas-controller-f6cdd9cc7-d2hzz" Jul 03 14:18:51.461450 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:51.461425 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwf6h\" (UniqueName: \"kubernetes.io/projected/be9798ef-bbf7-43e1-b485-ae851b3cdb96-kube-api-access-mwf6h\") pod \"maas-controller-f6cdd9cc7-d2hzz\" (UID: \"be9798ef-bbf7-43e1-b485-ae851b3cdb96\") " pod="opendatahub/maas-controller-f6cdd9cc7-d2hzz" Jul 03 14:18:51.539751 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:51.539723 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f6cdd9cc7-d2hzz" Jul 03 14:18:51.679431 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:51.679395 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f6cdd9cc7-d2hzz"] Jul 03 14:18:51.681292 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:18:51.681267 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9798ef_bbf7_43e1_b485_ae851b3cdb96.slice/crio-88fc7b0741b161b0ae35492b187b8d87572733de26413b23cbc130407eb552a1 WatchSource:0}: Error finding container 88fc7b0741b161b0ae35492b187b8d87572733de26413b23cbc130407eb552a1: Status 404 returned error can't find the container with id 88fc7b0741b161b0ae35492b187b8d87572733de26413b23cbc130407eb552a1 Jul 03 14:18:51.850015 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:51.849926 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f6cdd9cc7-d2hzz" event={"ID":"be9798ef-bbf7-43e1-b485-ae851b3cdb96","Type":"ContainerStarted","Data":"88fc7b0741b161b0ae35492b187b8d87572733de26413b23cbc130407eb552a1"} Jul 03 14:18:53.861957 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:53.861921 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f6cdd9cc7-d2hzz" event={"ID":"be9798ef-bbf7-43e1-b485-ae851b3cdb96","Type":"ContainerStarted","Data":"b2ae97f718409c5d429b388dcea89c4e442b113d6e4621bbb65ef8b03c4a8446"} Jul 03 14:18:53.862366 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:53.862066 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f6cdd9cc7-d2hzz" Jul 03 14:18:53.892429 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:18:53.892306 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f6cdd9cc7-d2hzz" podStartSLOduration=0.910117144 podStartE2EDuration="2.892286643s" totalImagesPullingTime="1.982169499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:18:53.891592704 +0000 UTC m=+612.247877325" watchObservedRunningTime="2026-07-03 14:18:53.892286643 +0000 UTC m=+612.248571265" Jul 03 14:19:04.849326 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:04.849254 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f6cdd9cc7-d2hzz"] Jul 03 14:19:04.849707 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:04.849455 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-f6cdd9cc7-d2hzz" podUID="be9798ef-bbf7-43e1-b485-ae851b3cdb96" containerName="manager" containerID="cri-o://b2ae97f718409c5d429b388dcea89c4e442b113d6e4621bbb65ef8b03c4a8446" gracePeriod=30 Jul 03 14:19:04.858807 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:04.858773 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f6cdd9cc7-d2hzz" Jul 03 14:19:05.091634 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:05.091608 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f6cdd9cc7-d2hzz" Jul 03 14:19:05.270844 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:05.270762 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be9798ef-bbf7-43e1-b485-ae851b3cdb96-webhook-cert\") pod \"be9798ef-bbf7-43e1-b485-ae851b3cdb96\" (UID: \"be9798ef-bbf7-43e1-b485-ae851b3cdb96\") " Jul 03 14:19:05.270844 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:05.270824 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwf6h\" (UniqueName: \"kubernetes.io/projected/be9798ef-bbf7-43e1-b485-ae851b3cdb96-kube-api-access-mwf6h\") pod \"be9798ef-bbf7-43e1-b485-ae851b3cdb96\" (UID: \"be9798ef-bbf7-43e1-b485-ae851b3cdb96\") " Jul 03 14:19:05.273233 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:05.273194 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9798ef-bbf7-43e1-b485-ae851b3cdb96-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "be9798ef-bbf7-43e1-b485-ae851b3cdb96" (UID: "be9798ef-bbf7-43e1-b485-ae851b3cdb96"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 14:19:05.273233 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:05.273210 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9798ef-bbf7-43e1-b485-ae851b3cdb96-kube-api-access-mwf6h" (OuterVolumeSpecName: "kube-api-access-mwf6h") pod "be9798ef-bbf7-43e1-b485-ae851b3cdb96" (UID: "be9798ef-bbf7-43e1-b485-ae851b3cdb96"). InnerVolumeSpecName "kube-api-access-mwf6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:19:05.371917 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:05.371868 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be9798ef-bbf7-43e1-b485-ae851b3cdb96-webhook-cert\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:19:05.371917 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:05.371910 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwf6h\" (UniqueName: \"kubernetes.io/projected/be9798ef-bbf7-43e1-b485-ae851b3cdb96-kube-api-access-mwf6h\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:19:05.913585 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:05.913549 2576 generic.go:356] "Generic (PLEG): container finished" podID="be9798ef-bbf7-43e1-b485-ae851b3cdb96" containerID="b2ae97f718409c5d429b388dcea89c4e442b113d6e4621bbb65ef8b03c4a8446" exitCode=0 Jul 03 14:19:05.914027 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:05.913594 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f6cdd9cc7-d2hzz" Jul 03 14:19:05.914027 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:05.913631 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f6cdd9cc7-d2hzz" event={"ID":"be9798ef-bbf7-43e1-b485-ae851b3cdb96","Type":"ContainerDied","Data":"b2ae97f718409c5d429b388dcea89c4e442b113d6e4621bbb65ef8b03c4a8446"} Jul 03 14:19:05.914027 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:05.913704 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f6cdd9cc7-d2hzz" event={"ID":"be9798ef-bbf7-43e1-b485-ae851b3cdb96","Type":"ContainerDied","Data":"88fc7b0741b161b0ae35492b187b8d87572733de26413b23cbc130407eb552a1"} Jul 03 14:19:05.914027 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:05.913728 2576 scope.go:117] "RemoveContainer" containerID="b2ae97f718409c5d429b388dcea89c4e442b113d6e4621bbb65ef8b03c4a8446" Jul 03 14:19:05.922321 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:05.922301 2576 scope.go:117] "RemoveContainer" containerID="b2ae97f718409c5d429b388dcea89c4e442b113d6e4621bbb65ef8b03c4a8446" Jul 03 14:19:05.922605 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:19:05.922585 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ae97f718409c5d429b388dcea89c4e442b113d6e4621bbb65ef8b03c4a8446\": container with ID starting with b2ae97f718409c5d429b388dcea89c4e442b113d6e4621bbb65ef8b03c4a8446 not found: ID does not exist" containerID="b2ae97f718409c5d429b388dcea89c4e442b113d6e4621bbb65ef8b03c4a8446" Jul 03 14:19:05.922674 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:05.922615 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ae97f718409c5d429b388dcea89c4e442b113d6e4621bbb65ef8b03c4a8446"} err="failed to get container status \"b2ae97f718409c5d429b388dcea89c4e442b113d6e4621bbb65ef8b03c4a8446\": rpc error: code = NotFound desc = could not find container \"b2ae97f718409c5d429b388dcea89c4e442b113d6e4621bbb65ef8b03c4a8446\": container with ID starting with b2ae97f718409c5d429b388dcea89c4e442b113d6e4621bbb65ef8b03c4a8446 not found: ID does not exist" Jul 03 14:19:05.938617 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:05.938594 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f6cdd9cc7-d2hzz"] Jul 03 14:19:05.943569 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:05.943545 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f6cdd9cc7-d2hzz"] Jul 03 14:19:06.168388 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:06.168311 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be9798ef-bbf7-43e1-b485-ae851b3cdb96" path="/var/lib/kubelet/pods/be9798ef-bbf7-43e1-b485-ae851b3cdb96/volumes" Jul 03 14:19:25.891618 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:25.891576 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-5b46495cf4-rhj5r"] Jul 03 14:19:25.892118 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:25.891903 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9798ef-bbf7-43e1-b485-ae851b3cdb96" containerName="manager" Jul 03 14:19:25.892118 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:25.891916 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9798ef-bbf7-43e1-b485-ae851b3cdb96" containerName="manager" Jul 03 14:19:25.892118 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:25.891976 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="be9798ef-bbf7-43e1-b485-ae851b3cdb96" containerName="manager" Jul 03 14:19:25.894829 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:25.894809 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5b46495cf4-rhj5r" Jul 03 14:19:25.898086 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:25.898067 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-f66fv\"" Jul 03 14:19:25.899192 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:25.899175 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 03 14:19:25.914635 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:25.914600 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-5b46495cf4-rhj5r"] Jul 03 14:19:25.920304 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:25.920278 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r82v8\" (UniqueName: \"kubernetes.io/projected/4bb76db1-1a34-47dc-9eb5-8624e53dd4e5-kube-api-access-r82v8\") pod \"maas-api-5b46495cf4-rhj5r\" (UID: \"4bb76db1-1a34-47dc-9eb5-8624e53dd4e5\") " pod="opendatahub/maas-api-5b46495cf4-rhj5r" Jul 03 14:19:25.920478 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:25.920309 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb76db1-1a34-47dc-9eb5-8624e53dd4e5-maas-api-tls\") pod \"maas-api-5b46495cf4-rhj5r\" (UID: \"4bb76db1-1a34-47dc-9eb5-8624e53dd4e5\") " pod="opendatahub/maas-api-5b46495cf4-rhj5r" Jul 03 14:19:25.920478 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:25.920332 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb76db1-1a34-47dc-9eb5-8624e53dd4e5-openshift-service-ca\") pod \"maas-api-5b46495cf4-rhj5r\" (UID: \"4bb76db1-1a34-47dc-9eb5-8624e53dd4e5\") " pod="opendatahub/maas-api-5b46495cf4-rhj5r" Jul 03 14:19:26.021763 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:26.021725 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r82v8\" (UniqueName: \"kubernetes.io/projected/4bb76db1-1a34-47dc-9eb5-8624e53dd4e5-kube-api-access-r82v8\") pod \"maas-api-5b46495cf4-rhj5r\" (UID: \"4bb76db1-1a34-47dc-9eb5-8624e53dd4e5\") " pod="opendatahub/maas-api-5b46495cf4-rhj5r" Jul 03 14:19:26.021971 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:26.021783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb76db1-1a34-47dc-9eb5-8624e53dd4e5-maas-api-tls\") pod \"maas-api-5b46495cf4-rhj5r\" (UID: \"4bb76db1-1a34-47dc-9eb5-8624e53dd4e5\") " pod="opendatahub/maas-api-5b46495cf4-rhj5r" Jul 03 14:19:26.021971 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:26.021824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb76db1-1a34-47dc-9eb5-8624e53dd4e5-openshift-service-ca\") pod \"maas-api-5b46495cf4-rhj5r\" (UID: \"4bb76db1-1a34-47dc-9eb5-8624e53dd4e5\") " pod="opendatahub/maas-api-5b46495cf4-rhj5r" Jul 03 14:19:26.021971 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:19:26.021937 2576 secret.go:189] Couldn't get secret opendatahub/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 03 14:19:26.022132 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:19:26.022018 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bb76db1-1a34-47dc-9eb5-8624e53dd4e5-maas-api-tls podName:4bb76db1-1a34-47dc-9eb5-8624e53dd4e5 nodeName:}" failed. No retries permitted until 2026-07-03 14:19:26.52199755 +0000 UTC m=+644.878282162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4bb76db1-1a34-47dc-9eb5-8624e53dd4e5-maas-api-tls") pod "maas-api-5b46495cf4-rhj5r" (UID: "4bb76db1-1a34-47dc-9eb5-8624e53dd4e5") : secret "maas-api-serving-cert" not found Jul 03 14:19:26.022622 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:26.022601 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb76db1-1a34-47dc-9eb5-8624e53dd4e5-openshift-service-ca\") pod \"maas-api-5b46495cf4-rhj5r\" (UID: \"4bb76db1-1a34-47dc-9eb5-8624e53dd4e5\") " pod="opendatahub/maas-api-5b46495cf4-rhj5r" Jul 03 14:19:26.035132 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:26.035108 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r82v8\" (UniqueName: \"kubernetes.io/projected/4bb76db1-1a34-47dc-9eb5-8624e53dd4e5-kube-api-access-r82v8\") pod \"maas-api-5b46495cf4-rhj5r\" (UID: \"4bb76db1-1a34-47dc-9eb5-8624e53dd4e5\") " pod="opendatahub/maas-api-5b46495cf4-rhj5r" Jul 03 14:19:26.525574 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:26.525532 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb76db1-1a34-47dc-9eb5-8624e53dd4e5-maas-api-tls\") pod \"maas-api-5b46495cf4-rhj5r\" (UID: \"4bb76db1-1a34-47dc-9eb5-8624e53dd4e5\") " pod="opendatahub/maas-api-5b46495cf4-rhj5r" Jul 03 14:19:26.528942 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:26.528917 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb76db1-1a34-47dc-9eb5-8624e53dd4e5-maas-api-tls\") pod \"maas-api-5b46495cf4-rhj5r\" (UID: \"4bb76db1-1a34-47dc-9eb5-8624e53dd4e5\") " pod="opendatahub/maas-api-5b46495cf4-rhj5r" Jul 03 14:19:26.804724 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:26.804694 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5b46495cf4-rhj5r" Jul 03 14:19:26.858610 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:26.858575 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-w69bs"] Jul 03 14:19:26.862475 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:26.862454 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-w69bs" Jul 03 14:19:26.867559 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:26.867230 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 03 14:19:26.867718 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:26.867599 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4n848\"" Jul 03 14:19:26.878619 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:26.878593 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-w69bs"] Jul 03 14:19:26.929866 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:26.929837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf13177-f69d-4de3-8f4f-93da301ec70a-plugins-config-volume\") pod \"payload-processing-58f86c7798-w69bs\" (UID: \"0bf13177-f69d-4de3-8f4f-93da301ec70a\") " pod="openshift-ingress/payload-processing-58f86c7798-w69bs" Jul 03 14:19:26.929866 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:26.929870 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlkbq\" (UniqueName: \"kubernetes.io/projected/0bf13177-f69d-4de3-8f4f-93da301ec70a-kube-api-access-wlkbq\") pod \"payload-processing-58f86c7798-w69bs\" (UID: \"0bf13177-f69d-4de3-8f4f-93da301ec70a\") " pod="openshift-ingress/payload-processing-58f86c7798-w69bs" Jul 03 14:19:26.965359 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:26.965336 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-5b46495cf4-rhj5r"] Jul 03 14:19:26.967164 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:19:26.967136 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb76db1_1a34_47dc_9eb5_8624e53dd4e5.slice/crio-302156b83bb65db22c19f18ebbc954339654b83bceeb8a987b30785afd18a1da WatchSource:0}: Error finding container 302156b83bb65db22c19f18ebbc954339654b83bceeb8a987b30785afd18a1da: Status 404 returned error can't find the container with id 302156b83bb65db22c19f18ebbc954339654b83bceeb8a987b30785afd18a1da Jul 03 14:19:26.968684 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:26.968638 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 03 14:19:27.009742 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:27.009706 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5b46495cf4-rhj5r" event={"ID":"4bb76db1-1a34-47dc-9eb5-8624e53dd4e5","Type":"ContainerStarted","Data":"302156b83bb65db22c19f18ebbc954339654b83bceeb8a987b30785afd18a1da"} Jul 03 14:19:27.031258 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:27.031226 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf13177-f69d-4de3-8f4f-93da301ec70a-plugins-config-volume\") pod \"payload-processing-58f86c7798-w69bs\" (UID: \"0bf13177-f69d-4de3-8f4f-93da301ec70a\") " pod="openshift-ingress/payload-processing-58f86c7798-w69bs" Jul 03 14:19:27.031258 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:27.031259 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlkbq\" (UniqueName: \"kubernetes.io/projected/0bf13177-f69d-4de3-8f4f-93da301ec70a-kube-api-access-wlkbq\") pod \"payload-processing-58f86c7798-w69bs\" (UID: \"0bf13177-f69d-4de3-8f4f-93da301ec70a\") " pod="openshift-ingress/payload-processing-58f86c7798-w69bs" Jul 03 14:19:27.032059 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:27.032041 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf13177-f69d-4de3-8f4f-93da301ec70a-plugins-config-volume\") pod \"payload-processing-58f86c7798-w69bs\" (UID: \"0bf13177-f69d-4de3-8f4f-93da301ec70a\") " pod="openshift-ingress/payload-processing-58f86c7798-w69bs" Jul 03 14:19:27.049220 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:27.049183 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlkbq\" (UniqueName: \"kubernetes.io/projected/0bf13177-f69d-4de3-8f4f-93da301ec70a-kube-api-access-wlkbq\") pod \"payload-processing-58f86c7798-w69bs\" (UID: \"0bf13177-f69d-4de3-8f4f-93da301ec70a\") " pod="openshift-ingress/payload-processing-58f86c7798-w69bs" Jul 03 14:19:27.179502 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:27.179411 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-w69bs" Jul 03 14:19:27.320952 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:27.320924 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-w69bs"] Jul 03 14:19:27.322176 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:19:27.322149 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf13177_f69d_4de3_8f4f_93da301ec70a.slice/crio-1e6ebce945fb8d05076c183ecc3434de16bbeeab38d18e31c5a846af41eaa9c7 WatchSource:0}: Error finding container 1e6ebce945fb8d05076c183ecc3434de16bbeeab38d18e31c5a846af41eaa9c7: Status 404 returned error can't find the container with id 1e6ebce945fb8d05076c183ecc3434de16bbeeab38d18e31c5a846af41eaa9c7 Jul 03 14:19:28.016242 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:28.016207 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-w69bs" event={"ID":"0bf13177-f69d-4de3-8f4f-93da301ec70a","Type":"ContainerStarted","Data":"1e6ebce945fb8d05076c183ecc3434de16bbeeab38d18e31c5a846af41eaa9c7"} Jul 03 14:19:29.022610 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:29.022571 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5b46495cf4-rhj5r" event={"ID":"4bb76db1-1a34-47dc-9eb5-8624e53dd4e5","Type":"ContainerStarted","Data":"1f93d371146ca421467a2f3266b4b9840aad4b70099168402825391021c1cf46"} Jul 03 14:19:29.023269 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:29.022797 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-5b46495cf4-rhj5r" Jul 03 14:19:29.024648 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:29.024614 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-w69bs" event={"ID":"0bf13177-f69d-4de3-8f4f-93da301ec70a","Type":"ContainerStarted","Data":"02c313c694cea7bf2333feb34ab522563499edacf68dc3c2b3a0057277e69066"} Jul 03 14:19:29.024802 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:29.024777 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-w69bs" Jul 03 14:19:29.050459 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:29.050358 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-5b46495cf4-rhj5r" podStartSLOduration=2.231106585 podStartE2EDuration="4.050342485s" totalImagesPullingTime="1.8192359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:19:29.048311203 +0000 UTC m=+647.404595823" watchObservedRunningTime="2026-07-03 14:19:29.050342485 +0000 UTC m=+647.406627155" Jul 03 14:19:29.073833 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:29.073777 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-w69bs" podStartSLOduration=1.6077282880000001 podStartE2EDuration="3.073760166s" totalImagesPullingTime="1.466031878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:19:29.072310564 +0000 UTC m=+647.428595202" watchObservedRunningTime="2026-07-03 14:19:29.073760166 +0000 UTC m=+647.430044787" Jul 03 14:19:35.034846 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:35.034811 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-5b46495cf4-rhj5r" Jul 03 14:19:40.031401 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:40.031374 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-w69bs" Jul 03 14:19:49.080349 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.080312 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz"] Jul 03 14:19:49.084710 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.084684 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.090307 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.090246 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 03 14:19:49.090455 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.090330 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xhx89\"" Jul 03 14:19:49.090989 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.090968 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 03 14:19:49.091767 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.091726 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 03 14:19:49.099760 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.099704 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz"] Jul 03 14:19:49.222705 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.222640 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ab82996-7cc1-431f-97fb-c7435fcb1d31-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vjvbz\" (UID: \"1ab82996-7cc1-431f-97fb-c7435fcb1d31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.222705 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.222701 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ab82996-7cc1-431f-97fb-c7435fcb1d31-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vjvbz\" (UID: \"1ab82996-7cc1-431f-97fb-c7435fcb1d31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.222959 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.222726 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ab82996-7cc1-431f-97fb-c7435fcb1d31-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vjvbz\" (UID: \"1ab82996-7cc1-431f-97fb-c7435fcb1d31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.222959 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.222744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ab82996-7cc1-431f-97fb-c7435fcb1d31-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vjvbz\" (UID: \"1ab82996-7cc1-431f-97fb-c7435fcb1d31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.222959 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.222775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ab82996-7cc1-431f-97fb-c7435fcb1d31-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vjvbz\" (UID: \"1ab82996-7cc1-431f-97fb-c7435fcb1d31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.222959 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.222835 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zqj9\" (UniqueName: \"kubernetes.io/projected/1ab82996-7cc1-431f-97fb-c7435fcb1d31-kube-api-access-9zqj9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vjvbz\" (UID: \"1ab82996-7cc1-431f-97fb-c7435fcb1d31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.323295 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.323256 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ab82996-7cc1-431f-97fb-c7435fcb1d31-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vjvbz\" (UID: \"1ab82996-7cc1-431f-97fb-c7435fcb1d31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.323503 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.323316 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zqj9\" (UniqueName: \"kubernetes.io/projected/1ab82996-7cc1-431f-97fb-c7435fcb1d31-kube-api-access-9zqj9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vjvbz\" (UID: \"1ab82996-7cc1-431f-97fb-c7435fcb1d31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.323503 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.323383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ab82996-7cc1-431f-97fb-c7435fcb1d31-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vjvbz\" (UID: \"1ab82996-7cc1-431f-97fb-c7435fcb1d31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.323503 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.323405 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ab82996-7cc1-431f-97fb-c7435fcb1d31-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vjvbz\" (UID: \"1ab82996-7cc1-431f-97fb-c7435fcb1d31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.323503 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.323441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ab82996-7cc1-431f-97fb-c7435fcb1d31-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vjvbz\" (UID: \"1ab82996-7cc1-431f-97fb-c7435fcb1d31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.323503 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.323465 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ab82996-7cc1-431f-97fb-c7435fcb1d31-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vjvbz\" (UID: \"1ab82996-7cc1-431f-97fb-c7435fcb1d31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.323788 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.323711 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ab82996-7cc1-431f-97fb-c7435fcb1d31-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vjvbz\" (UID: \"1ab82996-7cc1-431f-97fb-c7435fcb1d31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.323788 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.323754 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ab82996-7cc1-431f-97fb-c7435fcb1d31-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vjvbz\" (UID: \"1ab82996-7cc1-431f-97fb-c7435fcb1d31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.323895 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.323821 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ab82996-7cc1-431f-97fb-c7435fcb1d31-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vjvbz\" (UID: \"1ab82996-7cc1-431f-97fb-c7435fcb1d31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.325873 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.325849 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ab82996-7cc1-431f-97fb-c7435fcb1d31-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vjvbz\" (UID: \"1ab82996-7cc1-431f-97fb-c7435fcb1d31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.326055 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.326038 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ab82996-7cc1-431f-97fb-c7435fcb1d31-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vjvbz\" (UID: \"1ab82996-7cc1-431f-97fb-c7435fcb1d31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.334402 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.334351 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zqj9\" (UniqueName: \"kubernetes.io/projected/1ab82996-7cc1-431f-97fb-c7435fcb1d31-kube-api-access-9zqj9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vjvbz\" (UID: \"1ab82996-7cc1-431f-97fb-c7435fcb1d31\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.397595 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.397557 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:49.538607 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:49.538583 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz"] Jul 03 14:19:49.539901 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:19:49.539873 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab82996_7cc1_431f_97fb_c7435fcb1d31.slice/crio-a6b3c2b3fea7e8223d4f72821087c3d85d977ffb0abae3f5d412a7184eda766e WatchSource:0}: Error finding container a6b3c2b3fea7e8223d4f72821087c3d85d977ffb0abae3f5d412a7184eda766e: Status 404 returned error can't find the container with id a6b3c2b3fea7e8223d4f72821087c3d85d977ffb0abae3f5d412a7184eda766e Jul 03 14:19:50.123194 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:50.123150 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" event={"ID":"1ab82996-7cc1-431f-97fb-c7435fcb1d31","Type":"ContainerStarted","Data":"a6b3c2b3fea7e8223d4f72821087c3d85d977ffb0abae3f5d412a7184eda766e"} Jul 03 14:19:51.129884 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:51.129841 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" event={"ID":"1ab82996-7cc1-431f-97fb-c7435fcb1d31","Type":"ContainerStarted","Data":"4cea7271fb3e4beca7ced05a3cdf42838faf8d32a2e05dd198e0874f8f47d468"} Jul 03 14:19:51.152433 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:51.152385 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" podStartSLOduration=1.091681905 podStartE2EDuration="2.152368463s" totalImagesPullingTime="1.060686558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:19:51.150367243 +0000 UTC m=+669.506651864" watchObservedRunningTime="2026-07-03 14:19:51.152368463 +0000 UTC m=+669.508653105" Jul 03 14:19:59.398254 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:59.398213 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:59.398712 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:59.398565 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:19:59.410887 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:19:59.410866 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:20:00.182313 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:00.182283 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vjvbz" Jul 03 14:20:14.170785 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.170755 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s"] Jul 03 14:20:14.173120 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.173100 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.176276 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.176253 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 03 14:20:14.195106 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.195082 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s"] Jul 03 14:20:14.244115 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.244076 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bca895b8-a0a9-4211-86ba-8b5c956d1552-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rwf6s\" (UID: \"bca895b8-a0a9-4211-86ba-8b5c956d1552\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.244115 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.244110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bca895b8-a0a9-4211-86ba-8b5c956d1552-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rwf6s\" (UID: \"bca895b8-a0a9-4211-86ba-8b5c956d1552\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.244361 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.244131 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfjmp\" (UniqueName: \"kubernetes.io/projected/bca895b8-a0a9-4211-86ba-8b5c956d1552-kube-api-access-pfjmp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rwf6s\" (UID: \"bca895b8-a0a9-4211-86ba-8b5c956d1552\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.244361 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.244279 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca895b8-a0a9-4211-86ba-8b5c956d1552-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rwf6s\" (UID: \"bca895b8-a0a9-4211-86ba-8b5c956d1552\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.244473 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.244436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bca895b8-a0a9-4211-86ba-8b5c956d1552-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rwf6s\" (UID: \"bca895b8-a0a9-4211-86ba-8b5c956d1552\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.244473 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.244465 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca895b8-a0a9-4211-86ba-8b5c956d1552-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rwf6s\" (UID: \"bca895b8-a0a9-4211-86ba-8b5c956d1552\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.345224 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.345185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca895b8-a0a9-4211-86ba-8b5c956d1552-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rwf6s\" (UID: \"bca895b8-a0a9-4211-86ba-8b5c956d1552\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.345224 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.345229 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bca895b8-a0a9-4211-86ba-8b5c956d1552-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rwf6s\" (UID: \"bca895b8-a0a9-4211-86ba-8b5c956d1552\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.345467 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.345246 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca895b8-a0a9-4211-86ba-8b5c956d1552-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rwf6s\" (UID: \"bca895b8-a0a9-4211-86ba-8b5c956d1552\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.345467 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.345298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bca895b8-a0a9-4211-86ba-8b5c956d1552-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rwf6s\" (UID: \"bca895b8-a0a9-4211-86ba-8b5c956d1552\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.345467 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.345374 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bca895b8-a0a9-4211-86ba-8b5c956d1552-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rwf6s\" (UID: \"bca895b8-a0a9-4211-86ba-8b5c956d1552\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.345467 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.345410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfjmp\" (UniqueName: \"kubernetes.io/projected/bca895b8-a0a9-4211-86ba-8b5c956d1552-kube-api-access-pfjmp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rwf6s\" (UID: \"bca895b8-a0a9-4211-86ba-8b5c956d1552\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.345701 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.345633 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca895b8-a0a9-4211-86ba-8b5c956d1552-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rwf6s\" (UID: \"bca895b8-a0a9-4211-86ba-8b5c956d1552\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.345761 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.345739 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca895b8-a0a9-4211-86ba-8b5c956d1552-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rwf6s\" (UID: \"bca895b8-a0a9-4211-86ba-8b5c956d1552\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.345810 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.345779 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bca895b8-a0a9-4211-86ba-8b5c956d1552-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rwf6s\" (UID: \"bca895b8-a0a9-4211-86ba-8b5c956d1552\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.347726 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.347699 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bca895b8-a0a9-4211-86ba-8b5c956d1552-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rwf6s\" (UID: \"bca895b8-a0a9-4211-86ba-8b5c956d1552\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.347957 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.347938 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bca895b8-a0a9-4211-86ba-8b5c956d1552-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rwf6s\" (UID: \"bca895b8-a0a9-4211-86ba-8b5c956d1552\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.355293 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.355271 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfjmp\" (UniqueName: \"kubernetes.io/projected/bca895b8-a0a9-4211-86ba-8b5c956d1552-kube-api-access-pfjmp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rwf6s\" (UID: \"bca895b8-a0a9-4211-86ba-8b5c956d1552\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.483934 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.483846 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:14.622512 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:14.622469 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s"] Jul 03 14:20:14.626576 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:20:14.626546 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca895b8_a0a9_4211_86ba_8b5c956d1552.slice/crio-5c8a90a3900b272492152ade0ae25434181a62ab1c88829d2baf0320634274e5 WatchSource:0}: Error finding container 5c8a90a3900b272492152ade0ae25434181a62ab1c88829d2baf0320634274e5: Status 404 returned error can't find the container with id 5c8a90a3900b272492152ade0ae25434181a62ab1c88829d2baf0320634274e5 Jul 03 14:20:15.239016 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:15.238920 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" event={"ID":"bca895b8-a0a9-4211-86ba-8b5c956d1552","Type":"ContainerStarted","Data":"b6a52b675b708787c49a1cbd15d0011e48f113b52736dbe6d8088f6dab4b1666"} Jul 03 14:20:15.239016 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:15.238981 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" event={"ID":"bca895b8-a0a9-4211-86ba-8b5c956d1552","Type":"ContainerStarted","Data":"5c8a90a3900b272492152ade0ae25434181a62ab1c88829d2baf0320634274e5"} Jul 03 14:20:24.484995 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:24.484955 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:24.485485 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:24.485116 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:24.497443 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:24.497415 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:20:24.529806 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:24.529755 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" podStartSLOduration=10.343241258 podStartE2EDuration="10.529740725s" totalImagesPullingTime="186.499467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:20:15.265122334 +0000 UTC m=+693.621406953" watchObservedRunningTime="2026-07-03 14:20:24.529740725 +0000 UTC m=+702.886025396" Jul 03 14:20:25.296525 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:20:25.296499 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rwf6s" Jul 03 14:23:04.925460 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:04.925427 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bfc956fc6-5wsvb"] Jul 03 14:23:04.929072 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:04.929049 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bfc956fc6-5wsvb" Jul 03 14:23:04.933088 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:04.933067 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 03 14:23:04.934435 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:04.934412 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 03 14:23:04.934574 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:04.934452 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dlc6d\"" Jul 03 14:23:04.945851 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:04.945822 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bfc956fc6-5wsvb"] Jul 03 14:23:05.016044 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:05.016012 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e117128-6ed2-4f63-a796-7d4af29ddb63-webhook-cert\") pod \"maas-controller-7bfc956fc6-5wsvb\" (UID: \"1e117128-6ed2-4f63-a796-7d4af29ddb63\") " pod="opendatahub/maas-controller-7bfc956fc6-5wsvb" Jul 03 14:23:05.016224 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:05.016071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jhbm\" (UniqueName: \"kubernetes.io/projected/1e117128-6ed2-4f63-a796-7d4af29ddb63-kube-api-access-2jhbm\") pod \"maas-controller-7bfc956fc6-5wsvb\" (UID: \"1e117128-6ed2-4f63-a796-7d4af29ddb63\") " pod="opendatahub/maas-controller-7bfc956fc6-5wsvb" Jul 03 14:23:05.117143 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:05.117108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jhbm\" (UniqueName: \"kubernetes.io/projected/1e117128-6ed2-4f63-a796-7d4af29ddb63-kube-api-access-2jhbm\") pod \"maas-controller-7bfc956fc6-5wsvb\" (UID: \"1e117128-6ed2-4f63-a796-7d4af29ddb63\") " pod="opendatahub/maas-controller-7bfc956fc6-5wsvb" Jul 03 14:23:05.117310 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:05.117186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e117128-6ed2-4f63-a796-7d4af29ddb63-webhook-cert\") pod \"maas-controller-7bfc956fc6-5wsvb\" (UID: \"1e117128-6ed2-4f63-a796-7d4af29ddb63\") " pod="opendatahub/maas-controller-7bfc956fc6-5wsvb" Jul 03 14:23:05.119748 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:05.119717 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e117128-6ed2-4f63-a796-7d4af29ddb63-webhook-cert\") pod \"maas-controller-7bfc956fc6-5wsvb\" (UID: \"1e117128-6ed2-4f63-a796-7d4af29ddb63\") " pod="opendatahub/maas-controller-7bfc956fc6-5wsvb" Jul 03 14:23:05.128472 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:05.128447 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jhbm\" (UniqueName: \"kubernetes.io/projected/1e117128-6ed2-4f63-a796-7d4af29ddb63-kube-api-access-2jhbm\") pod \"maas-controller-7bfc956fc6-5wsvb\" (UID: \"1e117128-6ed2-4f63-a796-7d4af29ddb63\") " pod="opendatahub/maas-controller-7bfc956fc6-5wsvb" Jul 03 14:23:05.239514 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:05.239421 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bfc956fc6-5wsvb" Jul 03 14:23:05.384522 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:05.384492 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bfc956fc6-5wsvb"] Jul 03 14:23:05.386725 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:23:05.386698 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e117128_6ed2_4f63_a796_7d4af29ddb63.slice/crio-56780b24cd27bdd5c9b67066176dd610a1ffb2f4c606885666792251ce3b37f4 WatchSource:0}: Error finding container 56780b24cd27bdd5c9b67066176dd610a1ffb2f4c606885666792251ce3b37f4: Status 404 returned error can't find the container with id 56780b24cd27bdd5c9b67066176dd610a1ffb2f4c606885666792251ce3b37f4 Jul 03 14:23:06.015847 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:06.015808 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bfc956fc6-5wsvb" event={"ID":"1e117128-6ed2-4f63-a796-7d4af29ddb63","Type":"ContainerStarted","Data":"919df3b030c57746dc2475b184510e90bff6f3025c091d358f6b568e043d3f68"} Jul 03 14:23:06.016224 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:06.015863 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bfc956fc6-5wsvb" event={"ID":"1e117128-6ed2-4f63-a796-7d4af29ddb63","Type":"ContainerStarted","Data":"56780b24cd27bdd5c9b67066176dd610a1ffb2f4c606885666792251ce3b37f4"} Jul 03 14:23:06.016224 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:06.015947 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bfc956fc6-5wsvb" Jul 03 14:23:06.150743 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:06.150628 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7bfc956fc6-5wsvb" podStartSLOduration=1.733006435 podStartE2EDuration="2.150612348s" totalImagesPullingTime="417.605913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:23:06.146437916 +0000 UTC m=+864.502722558" watchObservedRunningTime="2026-07-03 14:23:06.150612348 +0000 UTC m=+864.506896968" Jul 03 14:23:17.033568 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:17.033535 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bfc956fc6-5wsvb" Jul 03 14:23:42.096962 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:42.096929 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:23:42.097542 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:42.097105 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:23:42.099896 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:42.099872 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:23:42.100018 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:23:42.099981 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:28:42.128752 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:28:42.128720 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:28:42.130250 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:28:42.130225 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:28:42.131801 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:28:42.131781 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:28:42.133257 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:28:42.133238 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:33:42.155288 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:33:42.155255 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:33:42.158080 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:33:42.158057 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:33:42.158478 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:33:42.158459 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:33:42.161265 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:33:42.161232 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:34:56.008548 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.008519 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr"] Jul 03 14:34:56.012126 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.012104 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.015377 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.015355 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-cb204bd5-openshift-default-dockercfg-prjbn\"" Jul 03 14:34:56.038718 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.038681 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr"] Jul 03 14:34:56.041103 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.041079 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-credential-socket\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.041259 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.041120 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-envoy\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.041259 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.041147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-token\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.041259 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.041206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istiod-ca-cert\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.041424 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.041260 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4znfm\" (UniqueName: \"kubernetes.io/projected/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-kube-api-access-4znfm\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.041424 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.041337 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-workload-socket\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.041424 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.041365 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-data\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.041424 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.041393 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-podinfo\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.041586 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.041461 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-workload-certs\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.142682 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.142617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-envoy\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.142682 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.142668 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-token\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.142938 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.142706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istiod-ca-cert\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.142938 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.142736 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4znfm\" (UniqueName: \"kubernetes.io/projected/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-kube-api-access-4znfm\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.142938 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.142777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-workload-socket\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.142938 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.142797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-data\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.142938 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.142829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-podinfo\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.142938 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.142913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-workload-certs\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.143255 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.142971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-credential-socket\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.143305 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.143259 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-workload-socket\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.143305 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.143295 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-credential-socket\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.143530 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.143505 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-data\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.143680 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.143640 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-workload-certs\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.143752 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.143678 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istiod-ca-cert\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.145198 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.145181 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-envoy\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.145564 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.145544 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-podinfo\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.155432 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.155406 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4znfm\" (UniqueName: \"kubernetes.io/projected/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-kube-api-access-4znfm\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.155557 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.155408 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-token\") pod \"e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.322902 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.322869 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:56.461459 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.461434 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr"] Jul 03 14:34:56.463577 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:34:56.463544 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407b7f6d_0b70_4eab_b5d3_7e94c540d0cd.slice/crio-961823e7cc73b066169c2f465f2bd7f278d5d875341259e87a88be3a4177f96f WatchSource:0}: Error finding container 961823e7cc73b066169c2f465f2bd7f278d5d875341259e87a88be3a4177f96f: Status 404 returned error can't find the container with id 961823e7cc73b066169c2f465f2bd7f278d5d875341259e87a88be3a4177f96f Jul 03 14:34:56.465602 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.465568 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:34:56.465718 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.465637 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:34:56.465718 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:56.465705 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:34:57.280201 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:57.280158 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" event={"ID":"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd","Type":"ContainerStarted","Data":"d348b6c7056e327b5e3899bee29c536ec1e0ff3887d58f44a8ffd7471a205169"} Jul 03 14:34:57.280609 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:57.280214 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" event={"ID":"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd","Type":"ContainerStarted","Data":"961823e7cc73b066169c2f465f2bd7f278d5d875341259e87a88be3a4177f96f"} Jul 03 14:34:57.298798 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:57.298767 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns"] Jul 03 14:34:57.302413 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:57.302389 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" Jul 03 14:34:57.305766 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:57.305744 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-cb204bd5-serving-cert\"" Jul 03 14:34:57.308986 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:57.308942 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" podStartSLOduration=2.308926606 podStartE2EDuration="2.308926606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:34:57.307107007 +0000 UTC m=+1575.663391629" watchObservedRunningTime="2026-07-03 14:34:57.308926606 +0000 UTC m=+1575.665211225" Jul 03 14:34:57.316570 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:57.316542 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns"] Jul 03 14:34:57.323986 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:57.323958 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:57.329305 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:57.329278 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:57.351377 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:57.351336 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s45v5\" (UniqueName: \"kubernetes.io/projected/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-kube-api-access-s45v5\") pod \"maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns\" (UID: \"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6\") " pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" Jul 03 14:34:57.351551 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:57.351391 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-openshift-service-ca\") pod \"maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns\" (UID: \"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6\") " pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" Jul 03 14:34:57.351551 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:57.351482 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-maas-api-tls\") pod \"maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns\" (UID: \"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6\") " pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" Jul 03 14:34:57.452373 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:57.452336 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-maas-api-tls\") pod \"maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns\" (UID: \"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6\") " pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" Jul 03 14:34:57.452584 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:57.452418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s45v5\" (UniqueName: \"kubernetes.io/projected/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-kube-api-access-s45v5\") pod \"maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns\" (UID: \"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6\") " pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" Jul 03 14:34:57.452584 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:57.452521 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-openshift-service-ca\") pod \"maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns\" (UID: \"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6\") " pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" Jul 03 14:34:57.452584 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:34:57.452530 2576 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-cb204bd5-serving-cert: secret "maas-api-e2e-ait-cb204bd5-serving-cert" not found Jul 03 14:34:57.452798 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:34:57.452603 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-maas-api-tls podName:42c69746-dfcb-40ec-b4f6-34ea85fc0ce6 nodeName:}" failed. No retries permitted until 2026-07-03 14:34:57.952581573 +0000 UTC m=+1576.308866177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-maas-api-tls") pod "maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" (UID: "42c69746-dfcb-40ec-b4f6-34ea85fc0ce6") : secret "maas-api-e2e-ait-cb204bd5-serving-cert" not found Jul 03 14:34:57.453382 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:57.453357 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-openshift-service-ca\") pod \"maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns\" (UID: \"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6\") " pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" Jul 03 14:34:57.464111 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:57.464084 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s45v5\" (UniqueName: \"kubernetes.io/projected/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-kube-api-access-s45v5\") pod \"maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns\" (UID: \"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6\") " pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" Jul 03 14:34:57.957647 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:57.957613 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-maas-api-tls\") pod \"maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns\" (UID: \"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6\") " pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" Jul 03 14:34:57.960200 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:57.960179 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-maas-api-tls\") pod \"maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns\" (UID: \"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6\") " pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" Jul 03 14:34:58.220258 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:58.220160 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" Jul 03 14:34:58.286582 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:58.285988 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:58.288284 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:58.287774 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:34:58.369963 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:58.369866 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns"] Jul 03 14:34:58.377874 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:34:58.377841 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c69746_dfcb_40ec_b4f6_34ea85fc0ce6.slice/crio-e7f776ed23eb40ea0e73403991b17e5befedbe9b5a547ef409d2aa906111167a WatchSource:0}: Error finding container e7f776ed23eb40ea0e73403991b17e5befedbe9b5a547ef409d2aa906111167a: Status 404 returned error can't find the container with id e7f776ed23eb40ea0e73403991b17e5befedbe9b5a547ef409d2aa906111167a Jul 03 14:34:58.379762 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:58.379742 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 03 14:34:59.292327 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:59.292284 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" event={"ID":"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6","Type":"ContainerStarted","Data":"65c9ac6a92f67cfb777e7cf535048045c09ae994a6da319db2c8d1c1ba345259"} Jul 03 14:34:59.292738 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:59.292495 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" Jul 03 14:34:59.292738 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:59.292527 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" event={"ID":"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6","Type":"ContainerStarted","Data":"e7f776ed23eb40ea0e73403991b17e5befedbe9b5a547ef409d2aa906111167a"} Jul 03 14:34:59.312816 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:34:59.312758 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" podStartSLOduration=1.804371553 podStartE2EDuration="2.312741737s" totalImagesPullingTime="508.370184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:34:59.311165124 +0000 UTC m=+1577.667449744" watchObservedRunningTime="2026-07-03 14:34:59.312741737 +0000 UTC m=+1577.669026358" Jul 03 14:35:02.595052 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:02.595018 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr"] Jul 03 14:35:02.595443 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:02.595196 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" podUID="407b7f6d-0b70-4eab-b5d3-7e94c540d0cd" containerName="istio-proxy" containerID="cri-o://d348b6c7056e327b5e3899bee29c536ec1e0ff3887d58f44a8ffd7471a205169" gracePeriod=30 Jul 03 14:35:02.793544 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:02.793509 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns"] Jul 03 14:35:02.793789 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:02.793761 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" podUID="42c69746-dfcb-40ec-b4f6-34ea85fc0ce6" containerName="maas-api" containerID="cri-o://65c9ac6a92f67cfb777e7cf535048045c09ae994a6da319db2c8d1c1ba345259" gracePeriod=30 Jul 03 14:35:03.103038 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:03.103010 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" Jul 03 14:35:03.206732 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:03.206627 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-openshift-service-ca\") pod \"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6\" (UID: \"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6\") " Jul 03 14:35:03.206880 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:03.206790 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s45v5\" (UniqueName: \"kubernetes.io/projected/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-kube-api-access-s45v5\") pod \"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6\" (UID: \"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6\") " Jul 03 14:35:03.206880 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:03.206873 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-maas-api-tls\") pod \"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6\" (UID: \"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6\") " Jul 03 14:35:03.207045 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:03.207023 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "42c69746-dfcb-40ec-b4f6-34ea85fc0ce6" (UID: "42c69746-dfcb-40ec-b4f6-34ea85fc0ce6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:35:03.207222 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:03.207200 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-openshift-service-ca\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:03.209065 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:03.209034 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "42c69746-dfcb-40ec-b4f6-34ea85fc0ce6" (UID: "42c69746-dfcb-40ec-b4f6-34ea85fc0ce6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 14:35:03.209166 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:03.209058 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-kube-api-access-s45v5" (OuterVolumeSpecName: "kube-api-access-s45v5") pod "42c69746-dfcb-40ec-b4f6-34ea85fc0ce6" (UID: "42c69746-dfcb-40ec-b4f6-34ea85fc0ce6"). InnerVolumeSpecName "kube-api-access-s45v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:35:03.308406 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:03.308371 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s45v5\" (UniqueName: \"kubernetes.io/projected/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-kube-api-access-s45v5\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:03.308406 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:03.308403 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6-maas-api-tls\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:03.313377 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:03.313340 2576 generic.go:356] "Generic (PLEG): container finished" podID="42c69746-dfcb-40ec-b4f6-34ea85fc0ce6" containerID="65c9ac6a92f67cfb777e7cf535048045c09ae994a6da319db2c8d1c1ba345259" exitCode=0 Jul 03 14:35:03.313377 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:03.313381 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" Jul 03 14:35:03.313602 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:03.313428 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" event={"ID":"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6","Type":"ContainerDied","Data":"65c9ac6a92f67cfb777e7cf535048045c09ae994a6da319db2c8d1c1ba345259"} Jul 03 14:35:03.313602 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:03.313475 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns" event={"ID":"42c69746-dfcb-40ec-b4f6-34ea85fc0ce6","Type":"ContainerDied","Data":"e7f776ed23eb40ea0e73403991b17e5befedbe9b5a547ef409d2aa906111167a"} Jul 03 14:35:03.313602 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:03.313496 2576 scope.go:117] "RemoveContainer" containerID="65c9ac6a92f67cfb777e7cf535048045c09ae994a6da319db2c8d1c1ba345259" Jul 03 14:35:03.324286 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:03.324259 2576 scope.go:117] "RemoveContainer" containerID="65c9ac6a92f67cfb777e7cf535048045c09ae994a6da319db2c8d1c1ba345259" Jul 03 14:35:03.324838 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:35:03.324805 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65c9ac6a92f67cfb777e7cf535048045c09ae994a6da319db2c8d1c1ba345259\": container with ID starting with 65c9ac6a92f67cfb777e7cf535048045c09ae994a6da319db2c8d1c1ba345259 not found: ID does not exist" containerID="65c9ac6a92f67cfb777e7cf535048045c09ae994a6da319db2c8d1c1ba345259" Jul 03 14:35:03.324953 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:03.324851 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65c9ac6a92f67cfb777e7cf535048045c09ae994a6da319db2c8d1c1ba345259"} err="failed to get container status \"65c9ac6a92f67cfb777e7cf535048045c09ae994a6da319db2c8d1c1ba345259\": rpc error: code = NotFound desc = could not find container \"65c9ac6a92f67cfb777e7cf535048045c09ae994a6da319db2c8d1c1ba345259\": container with ID starting with 65c9ac6a92f67cfb777e7cf535048045c09ae994a6da319db2c8d1c1ba345259 not found: ID does not exist" Jul 03 14:35:03.342635 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:03.342592 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns"] Jul 03 14:35:03.345851 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:03.345816 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-cb204bd5-7b574cdf49-zltns"] Jul 03 14:35:04.167898 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:04.167863 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42c69746-dfcb-40ec-b4f6-34ea85fc0ce6" path="/var/lib/kubelet/pods/42c69746-dfcb-40ec-b4f6-34ea85fc0ce6/volumes" Jul 03 14:35:07.759425 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.759400 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:35:07.853610 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.853518 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-podinfo\") pod \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " Jul 03 14:35:07.853610 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.853557 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istiod-ca-cert\") pod \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " Jul 03 14:35:07.853610 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.853592 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-token\") pod \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " Jul 03 14:35:07.853857 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.853647 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-envoy\") pod \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " Jul 03 14:35:07.853857 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.853712 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-data\") pod \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " Jul 03 14:35:07.853857 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.853733 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-workload-certs\") pod \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " Jul 03 14:35:07.853857 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.853757 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-credential-socket\") pod \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " Jul 03 14:35:07.853857 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.853806 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4znfm\" (UniqueName: \"kubernetes.io/projected/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-kube-api-access-4znfm\") pod \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " Jul 03 14:35:07.853857 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.853851 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-workload-socket\") pod \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\" (UID: \"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd\") " Jul 03 14:35:07.854139 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.854082 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "407b7f6d-0b70-4eab-b5d3-7e94c540d0cd" (UID: "407b7f6d-0b70-4eab-b5d3-7e94c540d0cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:35:07.854139 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.853985 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "407b7f6d-0b70-4eab-b5d3-7e94c540d0cd" (UID: "407b7f6d-0b70-4eab-b5d3-7e94c540d0cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:35:07.854139 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.854106 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "407b7f6d-0b70-4eab-b5d3-7e94c540d0cd" (UID: "407b7f6d-0b70-4eab-b5d3-7e94c540d0cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:35:07.854369 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.854325 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "407b7f6d-0b70-4eab-b5d3-7e94c540d0cd" (UID: "407b7f6d-0b70-4eab-b5d3-7e94c540d0cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:35:07.854719 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.854696 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-data" (OuterVolumeSpecName: "istio-data") pod "407b7f6d-0b70-4eab-b5d3-7e94c540d0cd" (UID: "407b7f6d-0b70-4eab-b5d3-7e94c540d0cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:35:07.856134 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.856109 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "407b7f6d-0b70-4eab-b5d3-7e94c540d0cd" (UID: "407b7f6d-0b70-4eab-b5d3-7e94c540d0cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:35:07.856225 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.856209 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-kube-api-access-4znfm" (OuterVolumeSpecName: "kube-api-access-4znfm") pod "407b7f6d-0b70-4eab-b5d3-7e94c540d0cd" (UID: "407b7f6d-0b70-4eab-b5d3-7e94c540d0cd"). InnerVolumeSpecName "kube-api-access-4znfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:35:07.856536 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.856511 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "407b7f6d-0b70-4eab-b5d3-7e94c540d0cd" (UID: "407b7f6d-0b70-4eab-b5d3-7e94c540d0cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 03 14:35:07.856536 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.856521 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-token" (OuterVolumeSpecName: "istio-token") pod "407b7f6d-0b70-4eab-b5d3-7e94c540d0cd" (UID: "407b7f6d-0b70-4eab-b5d3-7e94c540d0cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:35:07.955162 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.955127 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-envoy\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:07.955162 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.955156 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-data\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:07.955162 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.955165 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-workload-certs\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:07.955162 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.955174 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-credential-socket\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:07.955422 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.955184 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4znfm\" (UniqueName: \"kubernetes.io/projected/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-kube-api-access-4znfm\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:07.955422 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.955193 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-workload-socket\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:07.955422 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.955201 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-podinfo\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:07.955422 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.955210 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istiod-ca-cert\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:07.955422 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:07.955218 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd-istio-token\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:08.341238 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.341196 2576 generic.go:356] "Generic (PLEG): container finished" podID="407b7f6d-0b70-4eab-b5d3-7e94c540d0cd" containerID="d348b6c7056e327b5e3899bee29c536ec1e0ff3887d58f44a8ffd7471a205169" exitCode=0 Jul 03 14:35:08.341418 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.341244 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" event={"ID":"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd","Type":"ContainerDied","Data":"d348b6c7056e327b5e3899bee29c536ec1e0ff3887d58f44a8ffd7471a205169"} Jul 03 14:35:08.341418 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.341266 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" Jul 03 14:35:08.341418 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.341312 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr" event={"ID":"407b7f6d-0b70-4eab-b5d3-7e94c540d0cd","Type":"ContainerDied","Data":"961823e7cc73b066169c2f465f2bd7f278d5d875341259e87a88be3a4177f96f"} Jul 03 14:35:08.341418 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.341333 2576 scope.go:117] "RemoveContainer" containerID="d348b6c7056e327b5e3899bee29c536ec1e0ff3887d58f44a8ffd7471a205169" Jul 03 14:35:08.350840 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.350815 2576 scope.go:117] "RemoveContainer" containerID="d348b6c7056e327b5e3899bee29c536ec1e0ff3887d58f44a8ffd7471a205169" Jul 03 14:35:08.351129 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:35:08.351108 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d348b6c7056e327b5e3899bee29c536ec1e0ff3887d58f44a8ffd7471a205169\": container with ID starting with d348b6c7056e327b5e3899bee29c536ec1e0ff3887d58f44a8ffd7471a205169 not found: ID does not exist" containerID="d348b6c7056e327b5e3899bee29c536ec1e0ff3887d58f44a8ffd7471a205169" Jul 03 14:35:08.351180 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.351139 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d348b6c7056e327b5e3899bee29c536ec1e0ff3887d58f44a8ffd7471a205169"} err="failed to get container status \"d348b6c7056e327b5e3899bee29c536ec1e0ff3887d58f44a8ffd7471a205169\": rpc error: code = NotFound desc = could not find container \"d348b6c7056e327b5e3899bee29c536ec1e0ff3887d58f44a8ffd7471a205169\": container with ID starting with d348b6c7056e327b5e3899bee29c536ec1e0ff3887d58f44a8ffd7471a205169 not found: ID does not exist" Jul 03 14:35:08.360573 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.360545 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr"] Jul 03 14:35:08.367139 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.367109 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-cb204bd5-openshift-default-6f7bb4d4b7-6jjqr"] Jul 03 14:35:08.889777 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.889745 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5"] Jul 03 14:35:08.890200 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.890109 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="407b7f6d-0b70-4eab-b5d3-7e94c540d0cd" containerName="istio-proxy" Jul 03 14:35:08.890200 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.890126 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="407b7f6d-0b70-4eab-b5d3-7e94c540d0cd" containerName="istio-proxy" Jul 03 14:35:08.890200 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.890166 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42c69746-dfcb-40ec-b4f6-34ea85fc0ce6" containerName="maas-api" Jul 03 14:35:08.890200 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.890174 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="42c69746-dfcb-40ec-b4f6-34ea85fc0ce6" containerName="maas-api" Jul 03 14:35:08.890412 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.890250 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="407b7f6d-0b70-4eab-b5d3-7e94c540d0cd" containerName="istio-proxy" Jul 03 14:35:08.890412 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.890261 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="42c69746-dfcb-40ec-b4f6-34ea85fc0ce6" containerName="maas-api" Jul 03 14:35:08.894860 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.894837 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:08.902131 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.902103 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a28ab92b-openshift-default-dockercfg-m2tgx\"" Jul 03 14:35:08.919807 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.919775 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5"] Jul 03 14:35:08.965038 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.965001 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-credential-socket\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:08.965038 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.965037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13d5edb-3a21-42a6-9139-416df372f587-istiod-ca-cert\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:08.965269 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.965057 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c576p\" (UniqueName: \"kubernetes.io/projected/c13d5edb-3a21-42a6-9139-416df372f587-kube-api-access-c576p\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:08.965269 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.965105 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-workload-socket\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:08.965269 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.965122 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-workload-certs\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:08.965269 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.965167 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13d5edb-3a21-42a6-9139-416df372f587-istio-podinfo\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:08.965269 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.965186 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-istio-envoy\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:08.965269 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.965237 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13d5edb-3a21-42a6-9139-416df372f587-istio-token\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:08.965476 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:08.965281 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-istio-data\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.066714 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.066649 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-credential-socket\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.066714 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.066714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13d5edb-3a21-42a6-9139-416df372f587-istiod-ca-cert\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.066941 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.066734 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c576p\" (UniqueName: \"kubernetes.io/projected/c13d5edb-3a21-42a6-9139-416df372f587-kube-api-access-c576p\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.066941 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.066755 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-workload-socket\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.066941 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.066774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-workload-certs\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.066941 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.066817 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13d5edb-3a21-42a6-9139-416df372f587-istio-podinfo\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.066941 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.066836 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-istio-envoy\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.066941 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.066867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13d5edb-3a21-42a6-9139-416df372f587-istio-token\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.066941 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.066910 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-istio-data\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.067564 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.067123 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-credential-socket\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.067564 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.067159 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-workload-socket\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.067564 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.067496 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-workload-certs\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.067564 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.067526 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-istio-data\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.067845 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.067826 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13d5edb-3a21-42a6-9139-416df372f587-istiod-ca-cert\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.069488 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.069466 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-istio-envoy\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.069564 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.069502 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13d5edb-3a21-42a6-9139-416df372f587-istio-podinfo\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.081811 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.081784 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13d5edb-3a21-42a6-9139-416df372f587-istio-token\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.082022 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.081996 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c576p\" (UniqueName: \"kubernetes.io/projected/c13d5edb-3a21-42a6-9139-416df372f587-kube-api-access-c576p\") pod \"e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.206933 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.206835 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:09.352939 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.352915 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5"] Jul 03 14:35:09.354606 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:35:09.354577 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13d5edb_3a21_42a6_9139_416df372f587.slice/crio-6723a2bde1f4591fae158c9f27807be401e0c10e5184801f776fc31f7a3013ea WatchSource:0}: Error finding container 6723a2bde1f4591fae158c9f27807be401e0c10e5184801f776fc31f7a3013ea: Status 404 returned error can't find the container with id 6723a2bde1f4591fae158c9f27807be401e0c10e5184801f776fc31f7a3013ea Jul 03 14:35:09.356703 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.356648 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:35:09.356799 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.356751 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:35:09.356864 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:09.356798 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:35:10.168880 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:10.168844 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="407b7f6d-0b70-4eab-b5d3-7e94c540d0cd" path="/var/lib/kubelet/pods/407b7f6d-0b70-4eab-b5d3-7e94c540d0cd/volumes" Jul 03 14:35:10.270990 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:10.270949 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k"] Jul 03 14:35:10.274930 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:10.274907 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" Jul 03 14:35:10.278158 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:10.278131 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-a28ab92b-serving-cert\"" Jul 03 14:35:10.288609 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:10.288581 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k"] Jul 03 14:35:10.354899 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:10.354867 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" event={"ID":"c13d5edb-3a21-42a6-9139-416df372f587","Type":"ContainerStarted","Data":"9368531458d0e89dbd538ffd41bb524d6e2796c00a933a981f443a629b326830"} Jul 03 14:35:10.355078 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:10.354918 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" event={"ID":"c13d5edb-3a21-42a6-9139-416df372f587","Type":"ContainerStarted","Data":"6723a2bde1f4591fae158c9f27807be401e0c10e5184801f776fc31f7a3013ea"} Jul 03 14:35:10.378758 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:10.378710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4d66\" (UniqueName: \"kubernetes.io/projected/38f2109e-d976-4f64-bfae-515eb84fc229-kube-api-access-w4d66\") pod \"maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k\" (UID: \"38f2109e-d976-4f64-bfae-515eb84fc229\") " pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" Jul 03 14:35:10.378927 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:10.378782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f2109e-d976-4f64-bfae-515eb84fc229-maas-api-tls\") pod \"maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k\" (UID: \"38f2109e-d976-4f64-bfae-515eb84fc229\") " pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" Jul 03 14:35:10.378927 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:10.378884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f2109e-d976-4f64-bfae-515eb84fc229-openshift-service-ca\") pod \"maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k\" (UID: \"38f2109e-d976-4f64-bfae-515eb84fc229\") " pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" Jul 03 14:35:10.382747 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:10.382691 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" podStartSLOduration=2.382649804 podStartE2EDuration="2.382649804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:35:10.380554544 +0000 UTC m=+1588.736839166" watchObservedRunningTime="2026-07-03 14:35:10.382649804 +0000 UTC m=+1588.738934427" Jul 03 14:35:10.479933 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:10.479839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f2109e-d976-4f64-bfae-515eb84fc229-maas-api-tls\") pod \"maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k\" (UID: \"38f2109e-d976-4f64-bfae-515eb84fc229\") " pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" Jul 03 14:35:10.480105 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:10.480038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f2109e-d976-4f64-bfae-515eb84fc229-openshift-service-ca\") pod \"maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k\" (UID: \"38f2109e-d976-4f64-bfae-515eb84fc229\") " pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" Jul 03 14:35:10.480205 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:10.480156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4d66\" (UniqueName: \"kubernetes.io/projected/38f2109e-d976-4f64-bfae-515eb84fc229-kube-api-access-w4d66\") pod \"maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k\" (UID: \"38f2109e-d976-4f64-bfae-515eb84fc229\") " pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" Jul 03 14:35:10.480827 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:10.480755 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f2109e-d976-4f64-bfae-515eb84fc229-openshift-service-ca\") pod \"maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k\" (UID: \"38f2109e-d976-4f64-bfae-515eb84fc229\") " pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" Jul 03 14:35:10.482538 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:10.482518 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f2109e-d976-4f64-bfae-515eb84fc229-maas-api-tls\") pod \"maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k\" (UID: \"38f2109e-d976-4f64-bfae-515eb84fc229\") " pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" Jul 03 14:35:10.489237 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:10.489213 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4d66\" (UniqueName: \"kubernetes.io/projected/38f2109e-d976-4f64-bfae-515eb84fc229-kube-api-access-w4d66\") pod \"maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k\" (UID: \"38f2109e-d976-4f64-bfae-515eb84fc229\") " pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" Jul 03 14:35:10.586344 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:10.586310 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" Jul 03 14:35:10.734056 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:10.734029 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k"] Jul 03 14:35:10.735897 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:35:10.735871 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f2109e_d976_4f64_bfae_515eb84fc229.slice/crio-cdf87e8ab391a436484a0a07d84ba37c23a2609d35abd7dae2f46a005c67c000 WatchSource:0}: Error finding container cdf87e8ab391a436484a0a07d84ba37c23a2609d35abd7dae2f46a005c67c000: Status 404 returned error can't find the container with id cdf87e8ab391a436484a0a07d84ba37c23a2609d35abd7dae2f46a005c67c000 Jul 03 14:35:11.207487 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:11.207448 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:11.213393 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:11.213337 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:11.361900 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:11.361808 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" event={"ID":"38f2109e-d976-4f64-bfae-515eb84fc229","Type":"ContainerStarted","Data":"ec1fb92a4a8fa9cc88facd6a04afea02713e28abc2472df115d432bec576427c"} Jul 03 14:35:11.361900 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:11.361872 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" event={"ID":"38f2109e-d976-4f64-bfae-515eb84fc229","Type":"ContainerStarted","Data":"cdf87e8ab391a436484a0a07d84ba37c23a2609d35abd7dae2f46a005c67c000"} Jul 03 14:35:11.362100 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:11.362030 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:11.363142 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:11.363120 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:11.384177 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:11.384121 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" podStartSLOduration=1.050268141 podStartE2EDuration="1.384101114s" totalImagesPullingTime="333.832973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:35:11.382482407 +0000 UTC m=+1589.738767026" watchObservedRunningTime="2026-07-03 14:35:11.384101114 +0000 UTC m=+1589.740385739" Jul 03 14:35:12.367435 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:12.367406 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" Jul 03 14:35:15.543632 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:15.543585 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k"] Jul 03 14:35:15.545366 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:15.545330 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" podUID="38f2109e-d976-4f64-bfae-515eb84fc229" containerName="maas-api" containerID="cri-o://ec1fb92a4a8fa9cc88facd6a04afea02713e28abc2472df115d432bec576427c" gracePeriod=30 Jul 03 14:35:15.852889 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:15.852863 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" Jul 03 14:35:15.946048 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:15.946015 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f2109e-d976-4f64-bfae-515eb84fc229-openshift-service-ca\") pod \"38f2109e-d976-4f64-bfae-515eb84fc229\" (UID: \"38f2109e-d976-4f64-bfae-515eb84fc229\") " Jul 03 14:35:15.946259 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:15.946115 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4d66\" (UniqueName: \"kubernetes.io/projected/38f2109e-d976-4f64-bfae-515eb84fc229-kube-api-access-w4d66\") pod \"38f2109e-d976-4f64-bfae-515eb84fc229\" (UID: \"38f2109e-d976-4f64-bfae-515eb84fc229\") " Jul 03 14:35:15.946259 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:15.946150 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f2109e-d976-4f64-bfae-515eb84fc229-maas-api-tls\") pod \"38f2109e-d976-4f64-bfae-515eb84fc229\" (UID: \"38f2109e-d976-4f64-bfae-515eb84fc229\") " Jul 03 14:35:15.946418 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:15.946394 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f2109e-d976-4f64-bfae-515eb84fc229-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "38f2109e-d976-4f64-bfae-515eb84fc229" (UID: "38f2109e-d976-4f64-bfae-515eb84fc229"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:35:15.948431 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:15.948405 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f2109e-d976-4f64-bfae-515eb84fc229-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "38f2109e-d976-4f64-bfae-515eb84fc229" (UID: "38f2109e-d976-4f64-bfae-515eb84fc229"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 14:35:15.948551 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:15.948527 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f2109e-d976-4f64-bfae-515eb84fc229-kube-api-access-w4d66" (OuterVolumeSpecName: "kube-api-access-w4d66") pod "38f2109e-d976-4f64-bfae-515eb84fc229" (UID: "38f2109e-d976-4f64-bfae-515eb84fc229"). InnerVolumeSpecName "kube-api-access-w4d66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:35:16.047733 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:16.047692 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f2109e-d976-4f64-bfae-515eb84fc229-openshift-service-ca\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:16.047733 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:16.047726 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4d66\" (UniqueName: \"kubernetes.io/projected/38f2109e-d976-4f64-bfae-515eb84fc229-kube-api-access-w4d66\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:16.047733 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:16.047738 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f2109e-d976-4f64-bfae-515eb84fc229-maas-api-tls\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:16.385897 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:16.385856 2576 generic.go:356] "Generic (PLEG): container finished" podID="38f2109e-d976-4f64-bfae-515eb84fc229" containerID="ec1fb92a4a8fa9cc88facd6a04afea02713e28abc2472df115d432bec576427c" exitCode=0 Jul 03 14:35:16.386067 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:16.385923 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" Jul 03 14:35:16.386067 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:16.385940 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" event={"ID":"38f2109e-d976-4f64-bfae-515eb84fc229","Type":"ContainerDied","Data":"ec1fb92a4a8fa9cc88facd6a04afea02713e28abc2472df115d432bec576427c"} Jul 03 14:35:16.386067 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:16.386023 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k" event={"ID":"38f2109e-d976-4f64-bfae-515eb84fc229","Type":"ContainerDied","Data":"cdf87e8ab391a436484a0a07d84ba37c23a2609d35abd7dae2f46a005c67c000"} Jul 03 14:35:16.386067 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:16.386047 2576 scope.go:117] "RemoveContainer" containerID="ec1fb92a4a8fa9cc88facd6a04afea02713e28abc2472df115d432bec576427c" Jul 03 14:35:16.394688 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:16.394643 2576 scope.go:117] "RemoveContainer" containerID="ec1fb92a4a8fa9cc88facd6a04afea02713e28abc2472df115d432bec576427c" Jul 03 14:35:16.394996 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:35:16.394972 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec1fb92a4a8fa9cc88facd6a04afea02713e28abc2472df115d432bec576427c\": container with ID starting with ec1fb92a4a8fa9cc88facd6a04afea02713e28abc2472df115d432bec576427c not found: ID does not exist" containerID="ec1fb92a4a8fa9cc88facd6a04afea02713e28abc2472df115d432bec576427c" Jul 03 14:35:16.395054 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:16.395016 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1fb92a4a8fa9cc88facd6a04afea02713e28abc2472df115d432bec576427c"} err="failed to get container status \"ec1fb92a4a8fa9cc88facd6a04afea02713e28abc2472df115d432bec576427c\": rpc error: code = NotFound desc = could not find container \"ec1fb92a4a8fa9cc88facd6a04afea02713e28abc2472df115d432bec576427c\": container with ID starting with ec1fb92a4a8fa9cc88facd6a04afea02713e28abc2472df115d432bec576427c not found: ID does not exist" Jul 03 14:35:16.409931 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:16.409901 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k"] Jul 03 14:35:16.411850 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:16.411825 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-a28ab92b-58d466d544-wsr9k"] Jul 03 14:35:18.168411 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:18.168368 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38f2109e-d976-4f64-bfae-515eb84fc229" path="/var/lib/kubelet/pods/38f2109e-d976-4f64-bfae-515eb84fc229/volumes" Jul 03 14:35:21.481364 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:21.481326 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5"] Jul 03 14:35:21.481810 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:21.481580 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" podUID="c13d5edb-3a21-42a6-9139-416df372f587" containerName="istio-proxy" containerID="cri-o://9368531458d0e89dbd538ffd41bb524d6e2796c00a933a981f443a629b326830" gracePeriod=30 Jul 03 14:35:22.338178 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.338152 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:22.417694 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.417581 2576 generic.go:356] "Generic (PLEG): container finished" podID="c13d5edb-3a21-42a6-9139-416df372f587" containerID="9368531458d0e89dbd538ffd41bb524d6e2796c00a933a981f443a629b326830" exitCode=0 Jul 03 14:35:22.417864 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.417696 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" Jul 03 14:35:22.417864 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.417726 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" event={"ID":"c13d5edb-3a21-42a6-9139-416df372f587","Type":"ContainerDied","Data":"9368531458d0e89dbd538ffd41bb524d6e2796c00a933a981f443a629b326830"} Jul 03 14:35:22.417864 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.417794 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5" event={"ID":"c13d5edb-3a21-42a6-9139-416df372f587","Type":"ContainerDied","Data":"6723a2bde1f4591fae158c9f27807be401e0c10e5184801f776fc31f7a3013ea"} Jul 03 14:35:22.417864 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.417826 2576 scope.go:117] "RemoveContainer" containerID="9368531458d0e89dbd538ffd41bb524d6e2796c00a933a981f443a629b326830" Jul 03 14:35:22.426886 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.426813 2576 scope.go:117] "RemoveContainer" containerID="9368531458d0e89dbd538ffd41bb524d6e2796c00a933a981f443a629b326830" Jul 03 14:35:22.427191 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:35:22.427161 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9368531458d0e89dbd538ffd41bb524d6e2796c00a933a981f443a629b326830\": container with ID starting with 9368531458d0e89dbd538ffd41bb524d6e2796c00a933a981f443a629b326830 not found: ID does not exist" containerID="9368531458d0e89dbd538ffd41bb524d6e2796c00a933a981f443a629b326830" Jul 03 14:35:22.427295 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.427203 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9368531458d0e89dbd538ffd41bb524d6e2796c00a933a981f443a629b326830"} err="failed to get container status \"9368531458d0e89dbd538ffd41bb524d6e2796c00a933a981f443a629b326830\": rpc error: code = NotFound desc = could not find container \"9368531458d0e89dbd538ffd41bb524d6e2796c00a933a981f443a629b326830\": container with ID starting with 9368531458d0e89dbd538ffd41bb524d6e2796c00a933a981f443a629b326830 not found: ID does not exist" Jul 03 14:35:22.506829 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.506794 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-istio-envoy\") pod \"c13d5edb-3a21-42a6-9139-416df372f587\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " Jul 03 14:35:22.507302 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.506848 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13d5edb-3a21-42a6-9139-416df372f587-istio-podinfo\") pod \"c13d5edb-3a21-42a6-9139-416df372f587\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " Jul 03 14:35:22.507302 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.506889 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-credential-socket\") pod \"c13d5edb-3a21-42a6-9139-416df372f587\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " Jul 03 14:35:22.507302 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.507046 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-workload-certs\") pod \"c13d5edb-3a21-42a6-9139-416df372f587\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " Jul 03 14:35:22.507302 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.507091 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c576p\" (UniqueName: \"kubernetes.io/projected/c13d5edb-3a21-42a6-9139-416df372f587-kube-api-access-c576p\") pod \"c13d5edb-3a21-42a6-9139-416df372f587\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " Jul 03 14:35:22.507302 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.507120 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13d5edb-3a21-42a6-9139-416df372f587-istiod-ca-cert\") pod \"c13d5edb-3a21-42a6-9139-416df372f587\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " Jul 03 14:35:22.507302 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.507153 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c13d5edb-3a21-42a6-9139-416df372f587" (UID: "c13d5edb-3a21-42a6-9139-416df372f587"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:35:22.507302 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.507178 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13d5edb-3a21-42a6-9139-416df372f587-istio-token\") pod \"c13d5edb-3a21-42a6-9139-416df372f587\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " Jul 03 14:35:22.507302 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.507217 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-istio-data\") pod \"c13d5edb-3a21-42a6-9139-416df372f587\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " Jul 03 14:35:22.507302 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.507241 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-workload-socket\") pod \"c13d5edb-3a21-42a6-9139-416df372f587\" (UID: \"c13d5edb-3a21-42a6-9139-416df372f587\") " Jul 03 14:35:22.507834 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.507535 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-credential-socket\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:22.507834 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.507550 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13d5edb-3a21-42a6-9139-416df372f587-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c13d5edb-3a21-42a6-9139-416df372f587" (UID: "c13d5edb-3a21-42a6-9139-416df372f587"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:35:22.507834 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.507793 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c13d5edb-3a21-42a6-9139-416df372f587" (UID: "c13d5edb-3a21-42a6-9139-416df372f587"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:35:22.507834 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.507814 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c13d5edb-3a21-42a6-9139-416df372f587" (UID: "c13d5edb-3a21-42a6-9139-416df372f587"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:35:22.508825 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.508786 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-istio-data" (OuterVolumeSpecName: "istio-data") pod "c13d5edb-3a21-42a6-9139-416df372f587" (UID: "c13d5edb-3a21-42a6-9139-416df372f587"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:35:22.509466 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.509434 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c13d5edb-3a21-42a6-9139-416df372f587-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c13d5edb-3a21-42a6-9139-416df372f587" (UID: "c13d5edb-3a21-42a6-9139-416df372f587"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 03 14:35:22.509726 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.509705 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c13d5edb-3a21-42a6-9139-416df372f587" (UID: "c13d5edb-3a21-42a6-9139-416df372f587"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:35:22.509970 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.509950 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13d5edb-3a21-42a6-9139-416df372f587-kube-api-access-c576p" (OuterVolumeSpecName: "kube-api-access-c576p") pod "c13d5edb-3a21-42a6-9139-416df372f587" (UID: "c13d5edb-3a21-42a6-9139-416df372f587"). InnerVolumeSpecName "kube-api-access-c576p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:35:22.510115 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.510093 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13d5edb-3a21-42a6-9139-416df372f587-istio-token" (OuterVolumeSpecName: "istio-token") pod "c13d5edb-3a21-42a6-9139-416df372f587" (UID: "c13d5edb-3a21-42a6-9139-416df372f587"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:35:22.608467 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.608433 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-istio-envoy\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:22.608467 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.608464 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13d5edb-3a21-42a6-9139-416df372f587-istio-podinfo\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:22.608467 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.608472 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-workload-certs\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:22.608733 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.608481 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c576p\" (UniqueName: \"kubernetes.io/projected/c13d5edb-3a21-42a6-9139-416df372f587-kube-api-access-c576p\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:22.608733 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.608491 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13d5edb-3a21-42a6-9139-416df372f587-istiod-ca-cert\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:22.608733 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.608498 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13d5edb-3a21-42a6-9139-416df372f587-istio-token\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:22.608733 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.608506 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-istio-data\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:22.608733 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.608513 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13d5edb-3a21-42a6-9139-416df372f587-workload-socket\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:22.741406 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.741372 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5"] Jul 03 14:35:22.745267 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:22.745238 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a28ab92b-openshift-default-86c8fb5f97-wzvb5"] Jul 03 14:35:24.167907 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:24.167866 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c13d5edb-3a21-42a6-9139-416df372f587" path="/var/lib/kubelet/pods/c13d5edb-3a21-42a6-9139-416df372f587/volumes" Jul 03 14:35:27.568863 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.568822 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr"] Jul 03 14:35:27.569490 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.569183 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38f2109e-d976-4f64-bfae-515eb84fc229" containerName="maas-api" Jul 03 14:35:27.569490 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.569199 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="38f2109e-d976-4f64-bfae-515eb84fc229" containerName="maas-api" Jul 03 14:35:27.569490 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.569208 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13d5edb-3a21-42a6-9139-416df372f587" containerName="istio-proxy" Jul 03 14:35:27.569490 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.569214 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13d5edb-3a21-42a6-9139-416df372f587" containerName="istio-proxy" Jul 03 14:35:27.569490 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.569296 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13d5edb-3a21-42a6-9139-416df372f587" containerName="istio-proxy" Jul 03 14:35:27.569490 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.569307 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="38f2109e-d976-4f64-bfae-515eb84fc229" containerName="maas-api" Jul 03 14:35:27.572664 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.572631 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.575873 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.575845 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-cb4eb7a3-openshift-default-dockercfg-d5vcj\"" Jul 03 14:35:27.582787 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.582756 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr"] Jul 03 14:35:27.752344 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.752306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv8sq\" (UniqueName: \"kubernetes.io/projected/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-kube-api-access-tv8sq\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.752518 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.752354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-credential-socket\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.752518 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.752372 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-data\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.752518 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.752433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-workload-certs\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.752518 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.752478 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-workload-socket\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.752518 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.752496 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-envoy\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.752712 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.752574 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-podinfo\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.752712 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.752591 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istiod-ca-cert\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.752712 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.752608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-token\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.853484 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.853397 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-podinfo\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.853484 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.853438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istiod-ca-cert\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.853484 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.853458 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-token\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.853778 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.853505 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv8sq\" (UniqueName: \"kubernetes.io/projected/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-kube-api-access-tv8sq\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.853778 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.853535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-credential-socket\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.853778 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.853560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-data\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.853778 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.853588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-workload-certs\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.853778 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.853618 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-workload-socket\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.853778 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.853644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-envoy\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.854071 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.853983 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-data\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.854071 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.854043 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-credential-socket\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.854170 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.854146 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-workload-socket\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.854266 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.854248 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istiod-ca-cert\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.854478 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.854455 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-workload-certs\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.856062 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.856039 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-envoy\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.856241 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.856222 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-podinfo\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.862841 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.862815 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-token\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.862929 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.862854 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv8sq\" (UniqueName: \"kubernetes.io/projected/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-kube-api-access-tv8sq\") pod \"e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:27.886687 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:27.886639 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:28.028221 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:28.028169 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr"] Jul 03 14:35:28.030600 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:35:28.030572 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebeefb8c_3367_4af2_bf3f_4eae17c9dce6.slice/crio-e30166d1cf889765091eed9b322af2ee759d5b75ea9f2ee100d175baf5cd03fa WatchSource:0}: Error finding container e30166d1cf889765091eed9b322af2ee759d5b75ea9f2ee100d175baf5cd03fa: Status 404 returned error can't find the container with id e30166d1cf889765091eed9b322af2ee759d5b75ea9f2ee100d175baf5cd03fa Jul 03 14:35:28.033116 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:28.033072 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:35:28.033210 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:28.033154 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:35:28.033210 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:28.033186 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:35:28.452281 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:28.452240 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" event={"ID":"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6","Type":"ContainerStarted","Data":"53d5c08cece68bced91d79336a4813bda437859dcb50f7cbc2075c7476286122"} Jul 03 14:35:28.452483 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:28.452294 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" event={"ID":"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6","Type":"ContainerStarted","Data":"e30166d1cf889765091eed9b322af2ee759d5b75ea9f2ee100d175baf5cd03fa"} Jul 03 14:35:28.476008 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:28.475958 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" podStartSLOduration=1.475942538 podStartE2EDuration="1.475942538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:35:28.473421431 +0000 UTC m=+1606.829706053" watchObservedRunningTime="2026-07-03 14:35:28.475942538 +0000 UTC m=+1606.832227158" Jul 03 14:35:28.859494 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:28.859451 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6"] Jul 03 14:35:28.862549 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:28.862527 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" Jul 03 14:35:28.865490 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:28.865471 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-cb4eb7a3-serving-cert\"" Jul 03 14:35:28.874384 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:28.874358 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6"] Jul 03 14:35:28.887019 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:28.886986 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:28.964152 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:28.964100 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnl7r\" (UniqueName: \"kubernetes.io/projected/bd3bb242-5306-4d2f-8daf-0af750b2d553-kube-api-access-lnl7r\") pod \"maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6\" (UID: \"bd3bb242-5306-4d2f-8daf-0af750b2d553\") " pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" Jul 03 14:35:28.964345 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:28.964288 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3bb242-5306-4d2f-8daf-0af750b2d553-openshift-service-ca\") pod \"maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6\" (UID: \"bd3bb242-5306-4d2f-8daf-0af750b2d553\") " pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" Jul 03 14:35:28.964345 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:28.964327 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd3bb242-5306-4d2f-8daf-0af750b2d553-maas-api-tls\") pod \"maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6\" (UID: \"bd3bb242-5306-4d2f-8daf-0af750b2d553\") " pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" Jul 03 14:35:29.065626 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:29.065585 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3bb242-5306-4d2f-8daf-0af750b2d553-openshift-service-ca\") pod \"maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6\" (UID: \"bd3bb242-5306-4d2f-8daf-0af750b2d553\") " pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" Jul 03 14:35:29.065815 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:29.065634 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd3bb242-5306-4d2f-8daf-0af750b2d553-maas-api-tls\") pod \"maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6\" (UID: \"bd3bb242-5306-4d2f-8daf-0af750b2d553\") " pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" Jul 03 14:35:29.065815 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:29.065715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnl7r\" (UniqueName: \"kubernetes.io/projected/bd3bb242-5306-4d2f-8daf-0af750b2d553-kube-api-access-lnl7r\") pod \"maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6\" (UID: \"bd3bb242-5306-4d2f-8daf-0af750b2d553\") " pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" Jul 03 14:35:29.065895 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:35:29.065815 2576 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-cb4eb7a3-serving-cert: secret "maas-api-e2e-derive-cb4eb7a3-serving-cert" not found Jul 03 14:35:29.065895 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:35:29.065882 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3bb242-5306-4d2f-8daf-0af750b2d553-maas-api-tls podName:bd3bb242-5306-4d2f-8daf-0af750b2d553 nodeName:}" failed. No retries permitted until 2026-07-03 14:35:29.565865723 +0000 UTC m=+1607.922150320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd3bb242-5306-4d2f-8daf-0af750b2d553-maas-api-tls") pod "maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" (UID: "bd3bb242-5306-4d2f-8daf-0af750b2d553") : secret "maas-api-e2e-derive-cb4eb7a3-serving-cert" not found Jul 03 14:35:29.066431 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:29.066408 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3bb242-5306-4d2f-8daf-0af750b2d553-openshift-service-ca\") pod \"maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6\" (UID: \"bd3bb242-5306-4d2f-8daf-0af750b2d553\") " pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" Jul 03 14:35:29.075267 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:29.075242 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnl7r\" (UniqueName: \"kubernetes.io/projected/bd3bb242-5306-4d2f-8daf-0af750b2d553-kube-api-access-lnl7r\") pod \"maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6\" (UID: \"bd3bb242-5306-4d2f-8daf-0af750b2d553\") " pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" Jul 03 14:35:29.570209 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:29.570171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd3bb242-5306-4d2f-8daf-0af750b2d553-maas-api-tls\") pod \"maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6\" (UID: \"bd3bb242-5306-4d2f-8daf-0af750b2d553\") " pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" Jul 03 14:35:29.572708 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:29.572681 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd3bb242-5306-4d2f-8daf-0af750b2d553-maas-api-tls\") pod \"maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6\" (UID: \"bd3bb242-5306-4d2f-8daf-0af750b2d553\") " pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" Jul 03 14:35:29.776161 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:29.776125 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" Jul 03 14:35:29.893615 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:29.893582 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:29.913060 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:29.913025 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6"] Jul 03 14:35:29.915707 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:35:29.915649 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3bb242_5306_4d2f_8daf_0af750b2d553.slice/crio-0610ebe35c695bdd3ca4901bc948672f4eb36811d4246ab82305a2fb6fa37ed8 WatchSource:0}: Error finding container 0610ebe35c695bdd3ca4901bc948672f4eb36811d4246ab82305a2fb6fa37ed8: Status 404 returned error can't find the container with id 0610ebe35c695bdd3ca4901bc948672f4eb36811d4246ab82305a2fb6fa37ed8 Jul 03 14:35:30.464417 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:30.464379 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" event={"ID":"bd3bb242-5306-4d2f-8daf-0af750b2d553","Type":"ContainerStarted","Data":"3cff4d984c871595a0e61b4b127da0aeba64fe6f0cedcbaad16dee73339c7a47"} Jul 03 14:35:30.464605 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:30.464436 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" event={"ID":"bd3bb242-5306-4d2f-8daf-0af750b2d553","Type":"ContainerStarted","Data":"0610ebe35c695bdd3ca4901bc948672f4eb36811d4246ab82305a2fb6fa37ed8"} Jul 03 14:35:30.464717 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:30.464692 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:30.464773 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:30.464736 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" Jul 03 14:35:30.465584 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:30.465564 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:30.485552 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:30.485509 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" podStartSLOduration=2.081043696 podStartE2EDuration="2.485493922s" totalImagesPullingTime="404.450226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:35:30.483497737 +0000 UTC m=+1608.839782351" watchObservedRunningTime="2026-07-03 14:35:30.485493922 +0000 UTC m=+1608.841778541" Jul 03 14:35:33.436519 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:33.436409 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr"] Jul 03 14:35:33.479279 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:33.479236 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" podUID="ebeefb8c-3367-4af2-bf3f-4eae17c9dce6" containerName="istio-proxy" containerID="cri-o://53d5c08cece68bced91d79336a4813bda437859dcb50f7cbc2075c7476286122" gracePeriod=30 Jul 03 14:35:33.778826 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:33.778738 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6"] Jul 03 14:35:33.778970 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:33.778943 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" podUID="bd3bb242-5306-4d2f-8daf-0af750b2d553" containerName="maas-api" containerID="cri-o://3cff4d984c871595a0e61b4b127da0aeba64fe6f0cedcbaad16dee73339c7a47" gracePeriod=30 Jul 03 14:35:34.070468 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:34.070445 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" Jul 03 14:35:34.111746 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:34.111712 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnl7r\" (UniqueName: \"kubernetes.io/projected/bd3bb242-5306-4d2f-8daf-0af750b2d553-kube-api-access-lnl7r\") pod \"bd3bb242-5306-4d2f-8daf-0af750b2d553\" (UID: \"bd3bb242-5306-4d2f-8daf-0af750b2d553\") " Jul 03 14:35:34.111906 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:34.111757 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3bb242-5306-4d2f-8daf-0af750b2d553-openshift-service-ca\") pod \"bd3bb242-5306-4d2f-8daf-0af750b2d553\" (UID: \"bd3bb242-5306-4d2f-8daf-0af750b2d553\") " Jul 03 14:35:34.111906 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:34.111831 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd3bb242-5306-4d2f-8daf-0af750b2d553-maas-api-tls\") pod \"bd3bb242-5306-4d2f-8daf-0af750b2d553\" (UID: \"bd3bb242-5306-4d2f-8daf-0af750b2d553\") " Jul 03 14:35:34.112168 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:34.112143 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3bb242-5306-4d2f-8daf-0af750b2d553-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bd3bb242-5306-4d2f-8daf-0af750b2d553" (UID: "bd3bb242-5306-4d2f-8daf-0af750b2d553"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:35:34.114099 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:34.114068 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3bb242-5306-4d2f-8daf-0af750b2d553-kube-api-access-lnl7r" (OuterVolumeSpecName: "kube-api-access-lnl7r") pod "bd3bb242-5306-4d2f-8daf-0af750b2d553" (UID: "bd3bb242-5306-4d2f-8daf-0af750b2d553"). InnerVolumeSpecName "kube-api-access-lnl7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:35:34.114099 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:34.114069 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd3bb242-5306-4d2f-8daf-0af750b2d553-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bd3bb242-5306-4d2f-8daf-0af750b2d553" (UID: "bd3bb242-5306-4d2f-8daf-0af750b2d553"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 14:35:34.212859 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:34.212824 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd3bb242-5306-4d2f-8daf-0af750b2d553-maas-api-tls\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:34.212859 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:34.212856 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnl7r\" (UniqueName: \"kubernetes.io/projected/bd3bb242-5306-4d2f-8daf-0af750b2d553-kube-api-access-lnl7r\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:34.213048 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:34.212869 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3bb242-5306-4d2f-8daf-0af750b2d553-openshift-service-ca\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:34.485216 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:34.485116 2576 generic.go:356] "Generic (PLEG): container finished" podID="bd3bb242-5306-4d2f-8daf-0af750b2d553" containerID="3cff4d984c871595a0e61b4b127da0aeba64fe6f0cedcbaad16dee73339c7a47" exitCode=0 Jul 03 14:35:34.485216 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:34.485166 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" Jul 03 14:35:34.485750 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:34.485163 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" event={"ID":"bd3bb242-5306-4d2f-8daf-0af750b2d553","Type":"ContainerDied","Data":"3cff4d984c871595a0e61b4b127da0aeba64fe6f0cedcbaad16dee73339c7a47"} Jul 03 14:35:34.485750 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:34.485305 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6" event={"ID":"bd3bb242-5306-4d2f-8daf-0af750b2d553","Type":"ContainerDied","Data":"0610ebe35c695bdd3ca4901bc948672f4eb36811d4246ab82305a2fb6fa37ed8"} Jul 03 14:35:34.485750 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:34.485338 2576 scope.go:117] "RemoveContainer" containerID="3cff4d984c871595a0e61b4b127da0aeba64fe6f0cedcbaad16dee73339c7a47" Jul 03 14:35:34.493951 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:34.493927 2576 scope.go:117] "RemoveContainer" containerID="3cff4d984c871595a0e61b4b127da0aeba64fe6f0cedcbaad16dee73339c7a47" Jul 03 14:35:34.494222 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:35:34.494203 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cff4d984c871595a0e61b4b127da0aeba64fe6f0cedcbaad16dee73339c7a47\": container with ID starting with 3cff4d984c871595a0e61b4b127da0aeba64fe6f0cedcbaad16dee73339c7a47 not found: ID does not exist" containerID="3cff4d984c871595a0e61b4b127da0aeba64fe6f0cedcbaad16dee73339c7a47" Jul 03 14:35:34.494294 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:34.494230 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cff4d984c871595a0e61b4b127da0aeba64fe6f0cedcbaad16dee73339c7a47"} err="failed to get container status \"3cff4d984c871595a0e61b4b127da0aeba64fe6f0cedcbaad16dee73339c7a47\": rpc error: code = NotFound desc = could not find container \"3cff4d984c871595a0e61b4b127da0aeba64fe6f0cedcbaad16dee73339c7a47\": container with ID starting with 3cff4d984c871595a0e61b4b127da0aeba64fe6f0cedcbaad16dee73339c7a47 not found: ID does not exist" Jul 03 14:35:34.506951 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:34.506917 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6"] Jul 03 14:35:34.512495 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:34.512467 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-cb4eb7a3-59c95c86df-h5jx6"] Jul 03 14:35:36.168283 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:36.168197 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd3bb242-5306-4d2f-8daf-0af750b2d553" path="/var/lib/kubelet/pods/bd3bb242-5306-4d2f-8daf-0af750b2d553/volumes" Jul 03 14:35:39.429272 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.429248 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:39.475496 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.475459 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istiod-ca-cert\") pod \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " Jul 03 14:35:39.475679 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.475529 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-workload-socket\") pod \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " Jul 03 14:35:39.475679 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.475579 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-data\") pod \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " Jul 03 14:35:39.475679 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.475607 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-workload-certs\") pod \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " Jul 03 14:35:39.475679 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.475675 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-credential-socket\") pod \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " Jul 03 14:35:39.475885 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.475707 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-token\") pod \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " Jul 03 14:35:39.475885 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.475734 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-envoy\") pod \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " Jul 03 14:35:39.475885 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.475782 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tv8sq\" (UniqueName: \"kubernetes.io/projected/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-kube-api-access-tv8sq\") pod \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " Jul 03 14:35:39.475885 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.475818 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-podinfo\") pod \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\" (UID: \"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6\") " Jul 03 14:35:39.476082 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.475933 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ebeefb8c-3367-4af2-bf3f-4eae17c9dce6" (UID: "ebeefb8c-3367-4af2-bf3f-4eae17c9dce6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:35:39.476082 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.475999 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ebeefb8c-3367-4af2-bf3f-4eae17c9dce6" (UID: "ebeefb8c-3367-4af2-bf3f-4eae17c9dce6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:35:39.476196 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.476122 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ebeefb8c-3367-4af2-bf3f-4eae17c9dce6" (UID: "ebeefb8c-3367-4af2-bf3f-4eae17c9dce6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:35:39.476196 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.476170 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istiod-ca-cert\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:39.476196 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.476176 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ebeefb8c-3367-4af2-bf3f-4eae17c9dce6" (UID: "ebeefb8c-3367-4af2-bf3f-4eae17c9dce6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:35:39.476196 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.476189 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-workload-socket\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:39.476458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.476434 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-data" (OuterVolumeSpecName: "istio-data") pod "ebeefb8c-3367-4af2-bf3f-4eae17c9dce6" (UID: "ebeefb8c-3367-4af2-bf3f-4eae17c9dce6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:35:39.478337 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.478308 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-kube-api-access-tv8sq" (OuterVolumeSpecName: "kube-api-access-tv8sq") pod "ebeefb8c-3367-4af2-bf3f-4eae17c9dce6" (UID: "ebeefb8c-3367-4af2-bf3f-4eae17c9dce6"). InnerVolumeSpecName "kube-api-access-tv8sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:35:39.478579 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.478550 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-token" (OuterVolumeSpecName: "istio-token") pod "ebeefb8c-3367-4af2-bf3f-4eae17c9dce6" (UID: "ebeefb8c-3367-4af2-bf3f-4eae17c9dce6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:35:39.478673 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.478622 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ebeefb8c-3367-4af2-bf3f-4eae17c9dce6" (UID: "ebeefb8c-3367-4af2-bf3f-4eae17c9dce6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 03 14:35:39.478725 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.478681 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ebeefb8c-3367-4af2-bf3f-4eae17c9dce6" (UID: "ebeefb8c-3367-4af2-bf3f-4eae17c9dce6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:35:39.511453 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.511370 2576 generic.go:356] "Generic (PLEG): container finished" podID="ebeefb8c-3367-4af2-bf3f-4eae17c9dce6" containerID="53d5c08cece68bced91d79336a4813bda437859dcb50f7cbc2075c7476286122" exitCode=0 Jul 03 14:35:39.511453 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.511418 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" Jul 03 14:35:39.511453 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.511439 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" event={"ID":"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6","Type":"ContainerDied","Data":"53d5c08cece68bced91d79336a4813bda437859dcb50f7cbc2075c7476286122"} Jul 03 14:35:39.511665 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.511482 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr" event={"ID":"ebeefb8c-3367-4af2-bf3f-4eae17c9dce6","Type":"ContainerDied","Data":"e30166d1cf889765091eed9b322af2ee759d5b75ea9f2ee100d175baf5cd03fa"} Jul 03 14:35:39.511665 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.511504 2576 scope.go:117] "RemoveContainer" containerID="53d5c08cece68bced91d79336a4813bda437859dcb50f7cbc2075c7476286122" Jul 03 14:35:39.524897 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.524878 2576 scope.go:117] "RemoveContainer" containerID="53d5c08cece68bced91d79336a4813bda437859dcb50f7cbc2075c7476286122" Jul 03 14:35:39.525219 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:35:39.525199 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d5c08cece68bced91d79336a4813bda437859dcb50f7cbc2075c7476286122\": container with ID starting with 53d5c08cece68bced91d79336a4813bda437859dcb50f7cbc2075c7476286122 not found: ID does not exist" containerID="53d5c08cece68bced91d79336a4813bda437859dcb50f7cbc2075c7476286122" Jul 03 14:35:39.525270 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.525235 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d5c08cece68bced91d79336a4813bda437859dcb50f7cbc2075c7476286122"} err="failed to get container status \"53d5c08cece68bced91d79336a4813bda437859dcb50f7cbc2075c7476286122\": rpc error: code = NotFound desc = could not find container \"53d5c08cece68bced91d79336a4813bda437859dcb50f7cbc2075c7476286122\": container with ID starting with 53d5c08cece68bced91d79336a4813bda437859dcb50f7cbc2075c7476286122 not found: ID does not exist" Jul 03 14:35:39.535991 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.535958 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr"] Jul 03 14:35:39.540767 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.540738 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-cb4eb7a3-openshift-default-f78fc5675-t6hgr"] Jul 03 14:35:39.577519 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.577483 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-data\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:39.577519 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.577513 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-workload-certs\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:39.577519 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.577523 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-credential-socket\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:39.577776 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.577531 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-token\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:39.577776 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.577540 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-envoy\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:39.577776 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.577548 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv8sq\" (UniqueName: \"kubernetes.io/projected/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-kube-api-access-tv8sq\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:39.577776 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:39.577557 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6-istio-podinfo\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:35:40.168407 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:35:40.168364 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebeefb8c-3367-4af2-bf3f-4eae17c9dce6" path="/var/lib/kubelet/pods/ebeefb8c-3367-4af2-bf3f-4eae17c9dce6/volumes" Jul 03 14:37:15.239086 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.239007 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq"] Jul 03 14:37:15.239575 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.239346 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd3bb242-5306-4d2f-8daf-0af750b2d553" containerName="maas-api" Jul 03 14:37:15.239575 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.239356 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd3bb242-5306-4d2f-8daf-0af750b2d553" containerName="maas-api" Jul 03 14:37:15.239575 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.239367 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebeefb8c-3367-4af2-bf3f-4eae17c9dce6" containerName="istio-proxy" Jul 03 14:37:15.239575 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.239372 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebeefb8c-3367-4af2-bf3f-4eae17c9dce6" containerName="istio-proxy" Jul 03 14:37:15.239575 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.239436 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd3bb242-5306-4d2f-8daf-0af750b2d553" containerName="maas-api" Jul 03 14:37:15.239575 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.239445 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebeefb8c-3367-4af2-bf3f-4eae17c9dce6" containerName="istio-proxy" Jul 03 14:37:15.242492 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.242467 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.245317 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.245293 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-85fcff81-openshift-default-dockercfg-jsjdf\"" Jul 03 14:37:15.264758 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.264729 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq"] Jul 03 14:37:15.326530 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.326472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-workload-certs\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.326762 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.326538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937f8f84-52f1-469e-a535-89b625d32097-istio-podinfo\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.326762 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.326631 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937f8f84-52f1-469e-a535-89b625d32097-istio-token\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.326762 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.326674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937f8f84-52f1-469e-a535-89b625d32097-istiod-ca-cert\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.326762 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.326751 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-istio-envoy\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.326975 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.326773 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-istio-data\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.326975 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.326802 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-credential-socket\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.326975 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.326828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-workload-socket\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.326975 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.326898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwwkk\" (UniqueName: \"kubernetes.io/projected/937f8f84-52f1-469e-a535-89b625d32097-kube-api-access-mwwkk\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.428096 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.428049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937f8f84-52f1-469e-a535-89b625d32097-istio-token\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.428096 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.428096 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937f8f84-52f1-469e-a535-89b625d32097-istiod-ca-cert\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.428359 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.428166 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-istio-envoy\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.428359 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.428187 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-istio-data\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.428359 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.428212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-credential-socket\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.428507 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.428347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-workload-socket\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.428507 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.428403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwwkk\" (UniqueName: \"kubernetes.io/projected/937f8f84-52f1-469e-a535-89b625d32097-kube-api-access-mwwkk\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.428507 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.428442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-workload-certs\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.428507 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.428472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937f8f84-52f1-469e-a535-89b625d32097-istio-podinfo\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.428753 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.428559 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-istio-data\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.428997 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.428972 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937f8f84-52f1-469e-a535-89b625d32097-istiod-ca-cert\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.429098 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.429026 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-credential-socket\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.429098 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.429049 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-workload-certs\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.429302 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.429277 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-workload-socket\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.430768 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.430747 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-istio-envoy\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.431184 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.431162 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937f8f84-52f1-469e-a535-89b625d32097-istio-podinfo\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.442283 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.442251 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwwkk\" (UniqueName: \"kubernetes.io/projected/937f8f84-52f1-469e-a535-89b625d32097-kube-api-access-mwwkk\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.445177 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.445154 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937f8f84-52f1-469e-a535-89b625d32097-istio-token\") pod \"e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.554352 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.554324 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:15.701256 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.701230 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq"] Jul 03 14:37:15.702915 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:37:15.702878 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937f8f84_52f1_469e_a535_89b625d32097.slice/crio-cdc8f1179f62c8f6c541b05fd72f118c30ff1c1359e7eed4b89eba7513dcdf2e WatchSource:0}: Error finding container cdc8f1179f62c8f6c541b05fd72f118c30ff1c1359e7eed4b89eba7513dcdf2e: Status 404 returned error can't find the container with id cdc8f1179f62c8f6c541b05fd72f118c30ff1c1359e7eed4b89eba7513dcdf2e Jul 03 14:37:15.704933 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.704899 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:37:15.705012 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.704964 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:37:15.705012 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.704995 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:37:15.963001 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.962910 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" event={"ID":"937f8f84-52f1-469e-a535-89b625d32097","Type":"ContainerStarted","Data":"39bd0cc24d9037be2d35b352ffae5211e051ff6d48fb7d16776a4bdd6168cb0a"} Jul 03 14:37:15.963001 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.962974 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" event={"ID":"937f8f84-52f1-469e-a535-89b625d32097","Type":"ContainerStarted","Data":"cdc8f1179f62c8f6c541b05fd72f118c30ff1c1359e7eed4b89eba7513dcdf2e"} Jul 03 14:37:15.985539 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:15.985474 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" podStartSLOduration=0.985452669 podStartE2EDuration="985.452669ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:37:15.984860394 +0000 UTC m=+1714.341145015" watchObservedRunningTime="2026-07-03 14:37:15.985452669 +0000 UTC m=+1714.341737290" Jul 03 14:37:16.555206 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:16.555169 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:16.560085 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:16.560060 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:16.968045 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:16.967955 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:16.969072 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:16.969054 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:18.221765 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:18.221729 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr"] Jul 03 14:37:18.225575 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:18.225547 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" Jul 03 14:37:18.228719 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:18.228647 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-85fcff81-serving-cert\"" Jul 03 14:37:18.238454 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:18.238427 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr"] Jul 03 14:37:18.358013 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:18.357975 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c1ccb3-4b72-43f4-bee4-e25686a0235c-openshift-service-ca\") pod \"maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr\" (UID: \"68c1ccb3-4b72-43f4-bee4-e25686a0235c\") " pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" Jul 03 14:37:18.358184 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:18.358041 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68c1ccb3-4b72-43f4-bee4-e25686a0235c-maas-api-tls\") pod \"maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr\" (UID: \"68c1ccb3-4b72-43f4-bee4-e25686a0235c\") " pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" Jul 03 14:37:18.358184 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:18.358083 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shmgb\" (UniqueName: \"kubernetes.io/projected/68c1ccb3-4b72-43f4-bee4-e25686a0235c-kube-api-access-shmgb\") pod \"maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr\" (UID: \"68c1ccb3-4b72-43f4-bee4-e25686a0235c\") " pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" Jul 03 14:37:18.459288 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:18.459245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68c1ccb3-4b72-43f4-bee4-e25686a0235c-maas-api-tls\") pod \"maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr\" (UID: \"68c1ccb3-4b72-43f4-bee4-e25686a0235c\") " pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" Jul 03 14:37:18.459446 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:18.459313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shmgb\" (UniqueName: \"kubernetes.io/projected/68c1ccb3-4b72-43f4-bee4-e25686a0235c-kube-api-access-shmgb\") pod \"maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr\" (UID: \"68c1ccb3-4b72-43f4-bee4-e25686a0235c\") " pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" Jul 03 14:37:18.459446 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:18.459377 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c1ccb3-4b72-43f4-bee4-e25686a0235c-openshift-service-ca\") pod \"maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr\" (UID: \"68c1ccb3-4b72-43f4-bee4-e25686a0235c\") " pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" Jul 03 14:37:18.459446 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:37:18.459398 2576 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-85fcff81-serving-cert: secret "maas-api-e2e-mt-85fcff81-serving-cert" not found Jul 03 14:37:18.459599 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:37:18.459470 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68c1ccb3-4b72-43f4-bee4-e25686a0235c-maas-api-tls podName:68c1ccb3-4b72-43f4-bee4-e25686a0235c nodeName:}" failed. No retries permitted until 2026-07-03 14:37:18.959446092 +0000 UTC m=+1717.315730690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/68c1ccb3-4b72-43f4-bee4-e25686a0235c-maas-api-tls") pod "maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" (UID: "68c1ccb3-4b72-43f4-bee4-e25686a0235c") : secret "maas-api-e2e-mt-85fcff81-serving-cert" not found Jul 03 14:37:18.460165 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:18.460144 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c1ccb3-4b72-43f4-bee4-e25686a0235c-openshift-service-ca\") pod \"maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr\" (UID: \"68c1ccb3-4b72-43f4-bee4-e25686a0235c\") " pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" Jul 03 14:37:18.477759 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:18.477688 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shmgb\" (UniqueName: \"kubernetes.io/projected/68c1ccb3-4b72-43f4-bee4-e25686a0235c-kube-api-access-shmgb\") pod \"maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr\" (UID: \"68c1ccb3-4b72-43f4-bee4-e25686a0235c\") " pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" Jul 03 14:37:18.964638 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:18.964600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68c1ccb3-4b72-43f4-bee4-e25686a0235c-maas-api-tls\") pod \"maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr\" (UID: \"68c1ccb3-4b72-43f4-bee4-e25686a0235c\") " pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" Jul 03 14:37:18.967228 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:18.967199 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68c1ccb3-4b72-43f4-bee4-e25686a0235c-maas-api-tls\") pod \"maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr\" (UID: \"68c1ccb3-4b72-43f4-bee4-e25686a0235c\") " pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" Jul 03 14:37:19.138773 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:19.138739 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" Jul 03 14:37:19.283841 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:19.283809 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr"] Jul 03 14:37:19.285731 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:37:19.285705 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c1ccb3_4b72_43f4_bee4_e25686a0235c.slice/crio-ca5ff591f572e1ba377d89c5ca0894b35c0ed1df463dff265042eec65cb4e783 WatchSource:0}: Error finding container ca5ff591f572e1ba377d89c5ca0894b35c0ed1df463dff265042eec65cb4e783: Status 404 returned error can't find the container with id ca5ff591f572e1ba377d89c5ca0894b35c0ed1df463dff265042eec65cb4e783 Jul 03 14:37:19.983973 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:19.983931 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" event={"ID":"68c1ccb3-4b72-43f4-bee4-e25686a0235c","Type":"ContainerStarted","Data":"6bd46a216484e3c8524e0c700ef9b22e69041b08308b2e45620dec32a57aa7ba"} Jul 03 14:37:19.984162 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:19.983990 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" event={"ID":"68c1ccb3-4b72-43f4-bee4-e25686a0235c","Type":"ContainerStarted","Data":"ca5ff591f572e1ba377d89c5ca0894b35c0ed1df463dff265042eec65cb4e783"} Jul 03 14:37:19.984162 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:19.984042 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" Jul 03 14:37:20.003670 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:20.003603 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" podStartSLOduration=1.47804075 podStartE2EDuration="2.003580118s" totalImagesPullingTime="525.539368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:37:20.00224494 +0000 UTC m=+1718.358529560" watchObservedRunningTime="2026-07-03 14:37:20.003580118 +0000 UTC m=+1718.359864738" Jul 03 14:37:25.993914 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:25.993885 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" Jul 03 14:37:30.348276 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:30.348229 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr"] Jul 03 14:37:30.348964 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:30.348515 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" podUID="68c1ccb3-4b72-43f4-bee4-e25686a0235c" containerName="maas-api" containerID="cri-o://6bd46a216484e3c8524e0c700ef9b22e69041b08308b2e45620dec32a57aa7ba" gracePeriod=30 Jul 03 14:37:30.611805 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:30.611729 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" Jul 03 14:37:30.792955 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:30.792907 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68c1ccb3-4b72-43f4-bee4-e25686a0235c-maas-api-tls\") pod \"68c1ccb3-4b72-43f4-bee4-e25686a0235c\" (UID: \"68c1ccb3-4b72-43f4-bee4-e25686a0235c\") " Jul 03 14:37:30.793149 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:30.793009 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c1ccb3-4b72-43f4-bee4-e25686a0235c-openshift-service-ca\") pod \"68c1ccb3-4b72-43f4-bee4-e25686a0235c\" (UID: \"68c1ccb3-4b72-43f4-bee4-e25686a0235c\") " Jul 03 14:37:30.793149 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:30.793113 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shmgb\" (UniqueName: \"kubernetes.io/projected/68c1ccb3-4b72-43f4-bee4-e25686a0235c-kube-api-access-shmgb\") pod \"68c1ccb3-4b72-43f4-bee4-e25686a0235c\" (UID: \"68c1ccb3-4b72-43f4-bee4-e25686a0235c\") " Jul 03 14:37:30.793425 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:30.793396 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c1ccb3-4b72-43f4-bee4-e25686a0235c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68c1ccb3-4b72-43f4-bee4-e25686a0235c" (UID: "68c1ccb3-4b72-43f4-bee4-e25686a0235c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:37:30.795418 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:30.795392 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c1ccb3-4b72-43f4-bee4-e25686a0235c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68c1ccb3-4b72-43f4-bee4-e25686a0235c" (UID: "68c1ccb3-4b72-43f4-bee4-e25686a0235c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 14:37:30.795523 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:30.795472 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c1ccb3-4b72-43f4-bee4-e25686a0235c-kube-api-access-shmgb" (OuterVolumeSpecName: "kube-api-access-shmgb") pod "68c1ccb3-4b72-43f4-bee4-e25686a0235c" (UID: "68c1ccb3-4b72-43f4-bee4-e25686a0235c"). InnerVolumeSpecName "kube-api-access-shmgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:37:30.894255 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:30.894162 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shmgb\" (UniqueName: \"kubernetes.io/projected/68c1ccb3-4b72-43f4-bee4-e25686a0235c-kube-api-access-shmgb\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:37:30.894255 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:30.894201 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68c1ccb3-4b72-43f4-bee4-e25686a0235c-maas-api-tls\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:37:30.894255 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:30.894213 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c1ccb3-4b72-43f4-bee4-e25686a0235c-openshift-service-ca\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:37:31.054687 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:31.054612 2576 generic.go:356] "Generic (PLEG): container finished" podID="68c1ccb3-4b72-43f4-bee4-e25686a0235c" containerID="6bd46a216484e3c8524e0c700ef9b22e69041b08308b2e45620dec32a57aa7ba" exitCode=0 Jul 03 14:37:31.054687 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:31.054678 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" Jul 03 14:37:31.054910 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:31.054698 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" event={"ID":"68c1ccb3-4b72-43f4-bee4-e25686a0235c","Type":"ContainerDied","Data":"6bd46a216484e3c8524e0c700ef9b22e69041b08308b2e45620dec32a57aa7ba"} Jul 03 14:37:31.054910 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:31.054760 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr" event={"ID":"68c1ccb3-4b72-43f4-bee4-e25686a0235c","Type":"ContainerDied","Data":"ca5ff591f572e1ba377d89c5ca0894b35c0ed1df463dff265042eec65cb4e783"} Jul 03 14:37:31.054910 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:31.054791 2576 scope.go:117] "RemoveContainer" containerID="6bd46a216484e3c8524e0c700ef9b22e69041b08308b2e45620dec32a57aa7ba" Jul 03 14:37:31.079523 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:31.063979 2576 scope.go:117] "RemoveContainer" containerID="6bd46a216484e3c8524e0c700ef9b22e69041b08308b2e45620dec32a57aa7ba" Jul 03 14:37:31.079523 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:37:31.064295 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bd46a216484e3c8524e0c700ef9b22e69041b08308b2e45620dec32a57aa7ba\": container with ID starting with 6bd46a216484e3c8524e0c700ef9b22e69041b08308b2e45620dec32a57aa7ba not found: ID does not exist" containerID="6bd46a216484e3c8524e0c700ef9b22e69041b08308b2e45620dec32a57aa7ba" Jul 03 14:37:31.079523 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:31.064322 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bd46a216484e3c8524e0c700ef9b22e69041b08308b2e45620dec32a57aa7ba"} err="failed to get container status \"6bd46a216484e3c8524e0c700ef9b22e69041b08308b2e45620dec32a57aa7ba\": rpc error: code = NotFound desc = could not find container \"6bd46a216484e3c8524e0c700ef9b22e69041b08308b2e45620dec32a57aa7ba\": container with ID starting with 6bd46a216484e3c8524e0c700ef9b22e69041b08308b2e45620dec32a57aa7ba not found: ID does not exist" Jul 03 14:37:31.079523 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:31.078241 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr"] Jul 03 14:37:31.082766 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:31.082736 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-85fcff81-67cd49c9fc-6twrr"] Jul 03 14:37:31.969749 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:31.969712 2576 patch_prober.go:28] interesting pod/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 03 14:37:31.970145 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:31.969777 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" podUID="937f8f84-52f1-469e-a535-89b625d32097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 03 14:37:32.061825 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:32.061788 2576 generic.go:356] "Generic (PLEG): container finished" podID="937f8f84-52f1-469e-a535-89b625d32097" containerID="39bd0cc24d9037be2d35b352ffae5211e051ff6d48fb7d16776a4bdd6168cb0a" exitCode=0 Jul 03 14:37:32.061994 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:32.061858 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" event={"ID":"937f8f84-52f1-469e-a535-89b625d32097","Type":"ContainerDied","Data":"39bd0cc24d9037be2d35b352ffae5211e051ff6d48fb7d16776a4bdd6168cb0a"} Jul 03 14:37:32.062236 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:32.062223 2576 scope.go:117] "RemoveContainer" containerID="39bd0cc24d9037be2d35b352ffae5211e051ff6d48fb7d16776a4bdd6168cb0a" Jul 03 14:37:32.063290 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:32.063254 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:37:32.063394 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:32.063349 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:37:32.063394 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:32.063387 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:37:32.168636 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:32.168604 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68c1ccb3-4b72-43f4-bee4-e25686a0235c" path="/var/lib/kubelet/pods/68c1ccb3-4b72-43f4-bee4-e25686a0235c/volumes" Jul 03 14:37:33.068316 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:33.068283 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" event={"ID":"937f8f84-52f1-469e-a535-89b625d32097","Type":"ContainerStarted","Data":"f64076178e1512e710250e2e8c6c252c715457ac1f4520c86b33004d125c370d"} Jul 03 14:37:33.554622 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:33.554587 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:33.554835 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:33.554810 2576 patch_prober.go:28] interesting pod/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 03 14:37:33.554886 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:33.554867 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" podUID="937f8f84-52f1-469e-a535-89b625d32097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 03 14:37:34.555087 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:34.555044 2576 patch_prober.go:28] interesting pod/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 03 14:37:34.555495 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:34.555114 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" podUID="937f8f84-52f1-469e-a535-89b625d32097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 03 14:37:35.555558 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:35.555517 2576 patch_prober.go:28] interesting pod/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 03 14:37:35.556086 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:35.555584 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" podUID="937f8f84-52f1-469e-a535-89b625d32097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 03 14:37:36.555489 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:36.555453 2576 patch_prober.go:28] interesting pod/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 03 14:37:36.555751 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:36.555509 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" podUID="937f8f84-52f1-469e-a535-89b625d32097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 03 14:37:36.851490 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:36.851392 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq"] Jul 03 14:37:36.852791 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:36.852752 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" podUID="937f8f84-52f1-469e-a535-89b625d32097" containerName="istio-proxy" containerID="cri-o://f64076178e1512e710250e2e8c6c252c715457ac1f4520c86b33004d125c370d" gracePeriod=30 Jul 03 14:37:42.114346 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.114313 2576 generic.go:356] "Generic (PLEG): container finished" podID="937f8f84-52f1-469e-a535-89b625d32097" containerID="f64076178e1512e710250e2e8c6c252c715457ac1f4520c86b33004d125c370d" exitCode=0 Jul 03 14:37:42.114773 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.114396 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" event={"ID":"937f8f84-52f1-469e-a535-89b625d32097","Type":"ContainerDied","Data":"f64076178e1512e710250e2e8c6c252c715457ac1f4520c86b33004d125c370d"} Jul 03 14:37:42.114773 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.114505 2576 scope.go:117] "RemoveContainer" containerID="39bd0cc24d9037be2d35b352ffae5211e051ff6d48fb7d16776a4bdd6168cb0a" Jul 03 14:37:42.218123 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.218097 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:42.302218 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.302185 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-credential-socket\") pod \"937f8f84-52f1-469e-a535-89b625d32097\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " Jul 03 14:37:42.302218 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.302221 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-workload-socket\") pod \"937f8f84-52f1-469e-a535-89b625d32097\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " Jul 03 14:37:42.302440 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.302239 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937f8f84-52f1-469e-a535-89b625d32097-istio-token\") pod \"937f8f84-52f1-469e-a535-89b625d32097\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " Jul 03 14:37:42.302440 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.302280 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-workload-certs\") pod \"937f8f84-52f1-469e-a535-89b625d32097\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " Jul 03 14:37:42.302440 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.302321 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-istio-envoy\") pod \"937f8f84-52f1-469e-a535-89b625d32097\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " Jul 03 14:37:42.302440 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.302358 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937f8f84-52f1-469e-a535-89b625d32097-istiod-ca-cert\") pod \"937f8f84-52f1-469e-a535-89b625d32097\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " Jul 03 14:37:42.302440 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.302387 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-istio-data\") pod \"937f8f84-52f1-469e-a535-89b625d32097\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " Jul 03 14:37:42.302440 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.302413 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwwkk\" (UniqueName: \"kubernetes.io/projected/937f8f84-52f1-469e-a535-89b625d32097-kube-api-access-mwwkk\") pod \"937f8f84-52f1-469e-a535-89b625d32097\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " Jul 03 14:37:42.302779 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.302456 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937f8f84-52f1-469e-a535-89b625d32097-istio-podinfo\") pod \"937f8f84-52f1-469e-a535-89b625d32097\" (UID: \"937f8f84-52f1-469e-a535-89b625d32097\") " Jul 03 14:37:42.302779 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.302512 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "937f8f84-52f1-469e-a535-89b625d32097" (UID: "937f8f84-52f1-469e-a535-89b625d32097"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:37:42.302779 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.302549 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "937f8f84-52f1-469e-a535-89b625d32097" (UID: "937f8f84-52f1-469e-a535-89b625d32097"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:37:42.302779 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.302705 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "937f8f84-52f1-469e-a535-89b625d32097" (UID: "937f8f84-52f1-469e-a535-89b625d32097"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:37:42.302985 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.302808 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-credential-socket\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:37:42.302985 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.302816 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937f8f84-52f1-469e-a535-89b625d32097-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "937f8f84-52f1-469e-a535-89b625d32097" (UID: "937f8f84-52f1-469e-a535-89b625d32097"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:37:42.302985 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.302828 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-workload-socket\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:37:42.302985 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.302862 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-workload-certs\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:37:42.303404 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.303367 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-istio-data" (OuterVolumeSpecName: "istio-data") pod "937f8f84-52f1-469e-a535-89b625d32097" (UID: "937f8f84-52f1-469e-a535-89b625d32097"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:37:42.304852 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.304829 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/937f8f84-52f1-469e-a535-89b625d32097-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "937f8f84-52f1-469e-a535-89b625d32097" (UID: "937f8f84-52f1-469e-a535-89b625d32097"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 03 14:37:42.304942 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.304853 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937f8f84-52f1-469e-a535-89b625d32097-istio-token" (OuterVolumeSpecName: "istio-token") pod "937f8f84-52f1-469e-a535-89b625d32097" (UID: "937f8f84-52f1-469e-a535-89b625d32097"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:37:42.305120 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.305104 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937f8f84-52f1-469e-a535-89b625d32097-kube-api-access-mwwkk" (OuterVolumeSpecName: "kube-api-access-mwwkk") pod "937f8f84-52f1-469e-a535-89b625d32097" (UID: "937f8f84-52f1-469e-a535-89b625d32097"). InnerVolumeSpecName "kube-api-access-mwwkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:37:42.305161 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.305122 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "937f8f84-52f1-469e-a535-89b625d32097" (UID: "937f8f84-52f1-469e-a535-89b625d32097"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:37:42.403814 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.403775 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-istio-envoy\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:37:42.403814 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.403806 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937f8f84-52f1-469e-a535-89b625d32097-istiod-ca-cert\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:37:42.403814 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.403814 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937f8f84-52f1-469e-a535-89b625d32097-istio-data\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:37:42.404046 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.403823 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwwkk\" (UniqueName: \"kubernetes.io/projected/937f8f84-52f1-469e-a535-89b625d32097-kube-api-access-mwwkk\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:37:42.404046 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.403835 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937f8f84-52f1-469e-a535-89b625d32097-istio-podinfo\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:37:42.404046 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:42.403843 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937f8f84-52f1-469e-a535-89b625d32097-istio-token\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:37:43.120291 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:43.120259 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" event={"ID":"937f8f84-52f1-469e-a535-89b625d32097","Type":"ContainerDied","Data":"cdc8f1179f62c8f6c541b05fd72f118c30ff1c1359e7eed4b89eba7513dcdf2e"} Jul 03 14:37:43.120291 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:43.120281 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq" Jul 03 14:37:43.120809 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:43.120316 2576 scope.go:117] "RemoveContainer" containerID="f64076178e1512e710250e2e8c6c252c715457ac1f4520c86b33004d125c370d" Jul 03 14:37:43.142810 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:43.142775 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq"] Jul 03 14:37:43.147077 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:43.147042 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-85fcff81-openshift-default-7f5659cf9-9pvqq"] Jul 03 14:37:44.168304 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:44.168272 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="937f8f84-52f1-469e-a535-89b625d32097" path="/var/lib/kubelet/pods/937f8f84-52f1-469e-a535-89b625d32097/volumes" Jul 03 14:37:56.081727 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.081681 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47"] Jul 03 14:37:56.082217 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.082184 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937f8f84-52f1-469e-a535-89b625d32097" containerName="istio-proxy" Jul 03 14:37:56.082217 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.082203 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="937f8f84-52f1-469e-a535-89b625d32097" containerName="istio-proxy" Jul 03 14:37:56.082334 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.082240 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937f8f84-52f1-469e-a535-89b625d32097" containerName="istio-proxy" Jul 03 14:37:56.082334 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.082250 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="937f8f84-52f1-469e-a535-89b625d32097" containerName="istio-proxy" Jul 03 14:37:56.082334 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.082265 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68c1ccb3-4b72-43f4-bee4-e25686a0235c" containerName="maas-api" Jul 03 14:37:56.082334 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.082275 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="68c1ccb3-4b72-43f4-bee4-e25686a0235c" containerName="maas-api" Jul 03 14:37:56.082548 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.082384 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="68c1ccb3-4b72-43f4-bee4-e25686a0235c" containerName="maas-api" Jul 03 14:37:56.082548 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.082397 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="937f8f84-52f1-469e-a535-89b625d32097" containerName="istio-proxy" Jul 03 14:37:56.082548 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.082409 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="937f8f84-52f1-469e-a535-89b625d32097" containerName="istio-proxy" Jul 03 14:37:56.087318 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.087296 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.090181 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.090160 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-072d4360-openshift-default-dockercfg-cfjx4\"" Jul 03 14:37:56.096703 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.096675 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47"] Jul 03 14:37:56.227151 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.227114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-workload-socket\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.227151 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.227149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istiod-ca-cert\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.227354 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.227165 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6p2h\" (UniqueName: \"kubernetes.io/projected/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-kube-api-access-p6p2h\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.227354 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.227212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-workload-certs\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.227354 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.227254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-envoy\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.227354 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.227283 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-podinfo\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.227354 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.227297 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-token\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.227354 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.227313 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-data\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.227540 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.227399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-credential-socket\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.328410 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.328377 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-podinfo\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.328410 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.328411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-token\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.328687 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.328434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-data\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.328687 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.328499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-credential-socket\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.328687 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.328531 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-workload-socket\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.328687 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.328555 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istiod-ca-cert\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.328687 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.328576 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6p2h\" (UniqueName: \"kubernetes.io/projected/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-kube-api-access-p6p2h\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.328687 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.328621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-workload-certs\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.328687 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.328683 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-envoy\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.329049 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.329006 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-data\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.329049 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.329039 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-workload-socket\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.329154 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.329118 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-credential-socket\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.329195 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.329181 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-workload-certs\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.329701 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.329643 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istiod-ca-cert\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.331125 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.331102 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-envoy\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.331240 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.331220 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-podinfo\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.337070 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.337010 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-token\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.337070 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.337026 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6p2h\" (UniqueName: \"kubernetes.io/projected/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-kube-api-access-p6p2h\") pod \"e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.402230 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.402193 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:56.539401 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.539375 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47"] Jul 03 14:37:56.541288 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:37:56.541262 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c03b8d0_af95_4c62_8b9e_6dcd8b08aa84.slice/crio-c0b53103a26ff966cf78391431aba1b2d4c1802e61bfa04aeffb73cdbb5f0bca WatchSource:0}: Error finding container c0b53103a26ff966cf78391431aba1b2d4c1802e61bfa04aeffb73cdbb5f0bca: Status 404 returned error can't find the container with id c0b53103a26ff966cf78391431aba1b2d4c1802e61bfa04aeffb73cdbb5f0bca Jul 03 14:37:56.543407 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.543377 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:37:56.543464 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.543445 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:37:56.543503 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:56.543476 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:37:57.191772 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:57.191739 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" event={"ID":"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84","Type":"ContainerStarted","Data":"55819371fc3fcda761713f19bc6e866796e3c46315c2bed052a6b125aafe3365"} Jul 03 14:37:57.192149 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:57.191791 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" event={"ID":"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84","Type":"ContainerStarted","Data":"c0b53103a26ff966cf78391431aba1b2d4c1802e61bfa04aeffb73cdbb5f0bca"} Jul 03 14:37:57.216424 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:57.216373 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" podStartSLOduration=1.2163574050000001 podStartE2EDuration="1.216357405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:37:57.215220338 +0000 UTC m=+1755.571504957" watchObservedRunningTime="2026-07-03 14:37:57.216357405 +0000 UTC m=+1755.572642025" Jul 03 14:37:57.403030 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:57.402996 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:57.407840 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:57.407817 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:58.196353 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:58.196324 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:58.197314 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:58.197290 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:37:59.260411 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:59.260371 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml"] Jul 03 14:37:59.264470 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:59.264447 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" Jul 03 14:37:59.267382 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:59.267362 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-072d4360-serving-cert\"" Jul 03 14:37:59.274835 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:59.274810 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml"] Jul 03 14:37:59.358903 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:59.358867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4gnl\" (UniqueName: \"kubernetes.io/projected/8f534a30-6a94-4dda-9b9c-83e00c232b87-kube-api-access-j4gnl\") pod \"maas-api-e2e-mt-072d4360-6685d86688-n6zml\" (UID: \"8f534a30-6a94-4dda-9b9c-83e00c232b87\") " pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" Jul 03 14:37:59.359176 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:59.359145 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f534a30-6a94-4dda-9b9c-83e00c232b87-openshift-service-ca\") pod \"maas-api-e2e-mt-072d4360-6685d86688-n6zml\" (UID: \"8f534a30-6a94-4dda-9b9c-83e00c232b87\") " pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" Jul 03 14:37:59.359305 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:59.359285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f534a30-6a94-4dda-9b9c-83e00c232b87-maas-api-tls\") pod \"maas-api-e2e-mt-072d4360-6685d86688-n6zml\" (UID: \"8f534a30-6a94-4dda-9b9c-83e00c232b87\") " pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" Jul 03 14:37:59.460565 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:59.460530 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f534a30-6a94-4dda-9b9c-83e00c232b87-maas-api-tls\") pod \"maas-api-e2e-mt-072d4360-6685d86688-n6zml\" (UID: \"8f534a30-6a94-4dda-9b9c-83e00c232b87\") " pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" Jul 03 14:37:59.460810 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:59.460605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4gnl\" (UniqueName: \"kubernetes.io/projected/8f534a30-6a94-4dda-9b9c-83e00c232b87-kube-api-access-j4gnl\") pod \"maas-api-e2e-mt-072d4360-6685d86688-n6zml\" (UID: \"8f534a30-6a94-4dda-9b9c-83e00c232b87\") " pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" Jul 03 14:37:59.460810 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:59.460648 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f534a30-6a94-4dda-9b9c-83e00c232b87-openshift-service-ca\") pod \"maas-api-e2e-mt-072d4360-6685d86688-n6zml\" (UID: \"8f534a30-6a94-4dda-9b9c-83e00c232b87\") " pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" Jul 03 14:37:59.460810 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:37:59.460737 2576 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-072d4360-serving-cert: secret "maas-api-e2e-mt-072d4360-serving-cert" not found Jul 03 14:37:59.460982 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:37:59.460832 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f534a30-6a94-4dda-9b9c-83e00c232b87-maas-api-tls podName:8f534a30-6a94-4dda-9b9c-83e00c232b87 nodeName:}" failed. No retries permitted until 2026-07-03 14:37:59.960811031 +0000 UTC m=+1758.317095628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f534a30-6a94-4dda-9b9c-83e00c232b87-maas-api-tls") pod "maas-api-e2e-mt-072d4360-6685d86688-n6zml" (UID: "8f534a30-6a94-4dda-9b9c-83e00c232b87") : secret "maas-api-e2e-mt-072d4360-serving-cert" not found Jul 03 14:37:59.461272 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:59.461254 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f534a30-6a94-4dda-9b9c-83e00c232b87-openshift-service-ca\") pod \"maas-api-e2e-mt-072d4360-6685d86688-n6zml\" (UID: \"8f534a30-6a94-4dda-9b9c-83e00c232b87\") " pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" Jul 03 14:37:59.472463 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:59.472433 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4gnl\" (UniqueName: \"kubernetes.io/projected/8f534a30-6a94-4dda-9b9c-83e00c232b87-kube-api-access-j4gnl\") pod \"maas-api-e2e-mt-072d4360-6685d86688-n6zml\" (UID: \"8f534a30-6a94-4dda-9b9c-83e00c232b87\") " pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" Jul 03 14:37:59.966165 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:59.966127 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f534a30-6a94-4dda-9b9c-83e00c232b87-maas-api-tls\") pod \"maas-api-e2e-mt-072d4360-6685d86688-n6zml\" (UID: \"8f534a30-6a94-4dda-9b9c-83e00c232b87\") " pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" Jul 03 14:37:59.968696 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:37:59.968642 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f534a30-6a94-4dda-9b9c-83e00c232b87-maas-api-tls\") pod \"maas-api-e2e-mt-072d4360-6685d86688-n6zml\" (UID: \"8f534a30-6a94-4dda-9b9c-83e00c232b87\") " pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" Jul 03 14:38:00.179103 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:00.179063 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" Jul 03 14:38:00.318075 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:00.318049 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml"] Jul 03 14:38:00.319926 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:38:00.319883 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f534a30_6a94_4dda_9b9c_83e00c232b87.slice/crio-6db7dd85d239d8b8d7ad388b6cadb777386477946e7b2755eaf1515ae86d0915 WatchSource:0}: Error finding container 6db7dd85d239d8b8d7ad388b6cadb777386477946e7b2755eaf1515ae86d0915: Status 404 returned error can't find the container with id 6db7dd85d239d8b8d7ad388b6cadb777386477946e7b2755eaf1515ae86d0915 Jul 03 14:38:01.212750 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:01.212641 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" event={"ID":"8f534a30-6a94-4dda-9b9c-83e00c232b87","Type":"ContainerStarted","Data":"fa558ef4d68a4c3646f82d8e2932a6ac8095adf2b4d240c66235381923c8b175"} Jul 03 14:38:01.212750 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:01.212703 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" event={"ID":"8f534a30-6a94-4dda-9b9c-83e00c232b87","Type":"ContainerStarted","Data":"6db7dd85d239d8b8d7ad388b6cadb777386477946e7b2755eaf1515ae86d0915"} Jul 03 14:38:01.212750 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:01.212729 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" Jul 03 14:38:01.234788 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:01.234739 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" podStartSLOduration=1.700139142 podStartE2EDuration="2.234721666s" totalImagesPullingTime="534.582524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:38:01.233438086 +0000 UTC m=+1759.589722710" watchObservedRunningTime="2026-07-03 14:38:01.234721666 +0000 UTC m=+1759.591006285" Jul 03 14:38:07.230322 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:07.230291 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" Jul 03 14:38:13.197729 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:13.197698 2576 patch_prober.go:28] interesting pod/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 03 14:38:13.198099 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:13.197763 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" podUID="7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 03 14:38:13.277805 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:13.277769 2576 generic.go:356] "Generic (PLEG): container finished" podID="7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" containerID="55819371fc3fcda761713f19bc6e866796e3c46315c2bed052a6b125aafe3365" exitCode=0 Jul 03 14:38:13.277963 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:13.277816 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" event={"ID":"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84","Type":"ContainerDied","Data":"55819371fc3fcda761713f19bc6e866796e3c46315c2bed052a6b125aafe3365"} Jul 03 14:38:13.278178 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:13.278164 2576 scope.go:117] "RemoveContainer" containerID="55819371fc3fcda761713f19bc6e866796e3c46315c2bed052a6b125aafe3365" Jul 03 14:38:13.279177 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:13.279146 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:38:13.279229 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:13.279217 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:38:13.279264 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:13.279252 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:38:14.284338 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:14.284305 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" event={"ID":"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84","Type":"ContainerStarted","Data":"cd9b1e6396fd3967a639b6693b6cececcca45b9d8023e204c8dc76cc35a70177"} Jul 03 14:38:14.403076 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:14.403037 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:38:14.403271 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:14.403246 2576 patch_prober.go:28] interesting pod/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 03 14:38:14.403328 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:14.403308 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" podUID="7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 03 14:38:15.403553 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:15.403518 2576 patch_prober.go:28] interesting pod/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 03 14:38:15.403997 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:15.403577 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" podUID="7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 03 14:38:16.403016 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:16.402970 2576 patch_prober.go:28] interesting pod/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 03 14:38:16.403194 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:16.403038 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" podUID="7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 03 14:38:17.403720 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:17.403684 2576 patch_prober.go:28] interesting pod/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 03 14:38:17.404187 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:17.403748 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" podUID="7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 03 14:38:18.407442 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:18.407411 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:38:18.407903 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:18.407786 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:38:18.408404 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:18.408386 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:38:36.963038 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:36.962937 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml"] Jul 03 14:38:36.963524 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:36.963341 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" podUID="8f534a30-6a94-4dda-9b9c-83e00c232b87" containerName="maas-api" containerID="cri-o://fa558ef4d68a4c3646f82d8e2932a6ac8095adf2b4d240c66235381923c8b175" gracePeriod=30 Jul 03 14:38:37.204742 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:37.204717 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" Jul 03 14:38:37.322444 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:37.322392 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f534a30-6a94-4dda-9b9c-83e00c232b87-maas-api-tls\") pod \"8f534a30-6a94-4dda-9b9c-83e00c232b87\" (UID: \"8f534a30-6a94-4dda-9b9c-83e00c232b87\") " Jul 03 14:38:37.322644 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:37.322529 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4gnl\" (UniqueName: \"kubernetes.io/projected/8f534a30-6a94-4dda-9b9c-83e00c232b87-kube-api-access-j4gnl\") pod \"8f534a30-6a94-4dda-9b9c-83e00c232b87\" (UID: \"8f534a30-6a94-4dda-9b9c-83e00c232b87\") " Jul 03 14:38:37.322644 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:37.322562 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f534a30-6a94-4dda-9b9c-83e00c232b87-openshift-service-ca\") pod \"8f534a30-6a94-4dda-9b9c-83e00c232b87\" (UID: \"8f534a30-6a94-4dda-9b9c-83e00c232b87\") " Jul 03 14:38:37.322999 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:37.322968 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f534a30-6a94-4dda-9b9c-83e00c232b87-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8f534a30-6a94-4dda-9b9c-83e00c232b87" (UID: "8f534a30-6a94-4dda-9b9c-83e00c232b87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:38:37.324727 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:37.324696 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f534a30-6a94-4dda-9b9c-83e00c232b87-kube-api-access-j4gnl" (OuterVolumeSpecName: "kube-api-access-j4gnl") pod "8f534a30-6a94-4dda-9b9c-83e00c232b87" (UID: "8f534a30-6a94-4dda-9b9c-83e00c232b87"). InnerVolumeSpecName "kube-api-access-j4gnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:38:37.324825 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:37.324748 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f534a30-6a94-4dda-9b9c-83e00c232b87-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8f534a30-6a94-4dda-9b9c-83e00c232b87" (UID: "8f534a30-6a94-4dda-9b9c-83e00c232b87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 14:38:37.386594 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:37.386557 2576 generic.go:356] "Generic (PLEG): container finished" podID="8f534a30-6a94-4dda-9b9c-83e00c232b87" containerID="fa558ef4d68a4c3646f82d8e2932a6ac8095adf2b4d240c66235381923c8b175" exitCode=0 Jul 03 14:38:37.386770 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:37.386604 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" Jul 03 14:38:37.386770 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:37.386613 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" event={"ID":"8f534a30-6a94-4dda-9b9c-83e00c232b87","Type":"ContainerDied","Data":"fa558ef4d68a4c3646f82d8e2932a6ac8095adf2b4d240c66235381923c8b175"} Jul 03 14:38:37.386770 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:37.386687 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml" event={"ID":"8f534a30-6a94-4dda-9b9c-83e00c232b87","Type":"ContainerDied","Data":"6db7dd85d239d8b8d7ad388b6cadb777386477946e7b2755eaf1515ae86d0915"} Jul 03 14:38:37.386770 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:37.386712 2576 scope.go:117] "RemoveContainer" containerID="fa558ef4d68a4c3646f82d8e2932a6ac8095adf2b4d240c66235381923c8b175" Jul 03 14:38:37.397104 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:37.397082 2576 scope.go:117] "RemoveContainer" containerID="fa558ef4d68a4c3646f82d8e2932a6ac8095adf2b4d240c66235381923c8b175" Jul 03 14:38:37.397378 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:38:37.397359 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa558ef4d68a4c3646f82d8e2932a6ac8095adf2b4d240c66235381923c8b175\": container with ID starting with fa558ef4d68a4c3646f82d8e2932a6ac8095adf2b4d240c66235381923c8b175 not found: ID does not exist" containerID="fa558ef4d68a4c3646f82d8e2932a6ac8095adf2b4d240c66235381923c8b175" Jul 03 14:38:37.397446 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:37.397386 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa558ef4d68a4c3646f82d8e2932a6ac8095adf2b4d240c66235381923c8b175"} err="failed to get container status \"fa558ef4d68a4c3646f82d8e2932a6ac8095adf2b4d240c66235381923c8b175\": rpc error: code = NotFound desc = could not find container \"fa558ef4d68a4c3646f82d8e2932a6ac8095adf2b4d240c66235381923c8b175\": container with ID starting with fa558ef4d68a4c3646f82d8e2932a6ac8095adf2b4d240c66235381923c8b175 not found: ID does not exist" Jul 03 14:38:37.411731 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:37.411705 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml"] Jul 03 14:38:37.416213 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:37.416186 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-072d4360-6685d86688-n6zml"] Jul 03 14:38:37.424006 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:37.423973 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4gnl\" (UniqueName: \"kubernetes.io/projected/8f534a30-6a94-4dda-9b9c-83e00c232b87-kube-api-access-j4gnl\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:38:37.424006 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:37.424009 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f534a30-6a94-4dda-9b9c-83e00c232b87-openshift-service-ca\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:38:37.424122 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:37.424019 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f534a30-6a94-4dda-9b9c-83e00c232b87-maas-api-tls\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:38:38.168582 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:38.168548 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f534a30-6a94-4dda-9b9c-83e00c232b87" path="/var/lib/kubelet/pods/8f534a30-6a94-4dda-9b9c-83e00c232b87/volumes" Jul 03 14:38:42.185405 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:42.185375 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:38:42.188447 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:42.188424 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:38:42.189321 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:42.189299 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:38:42.192072 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:38:42.192054 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:39:30.510852 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:30.510809 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47"] Jul 03 14:39:30.511420 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:30.511041 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" podUID="7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" containerName="istio-proxy" containerID="cri-o://cd9b1e6396fd3967a639b6693b6cececcca45b9d8023e204c8dc76cc35a70177" gracePeriod=30 Jul 03 14:39:33.409618 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:33.409571 2576 patch_prober.go:28] interesting pod/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 03 14:39:33.410025 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:33.409640 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" podUID="7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 03 14:39:35.660204 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.660170 2576 generic.go:356] "Generic (PLEG): container finished" podID="7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" containerID="cd9b1e6396fd3967a639b6693b6cececcca45b9d8023e204c8dc76cc35a70177" exitCode=0 Jul 03 14:39:35.660626 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.660248 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" event={"ID":"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84","Type":"ContainerDied","Data":"cd9b1e6396fd3967a639b6693b6cececcca45b9d8023e204c8dc76cc35a70177"} Jul 03 14:39:35.660626 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.660312 2576 scope.go:117] "RemoveContainer" containerID="55819371fc3fcda761713f19bc6e866796e3c46315c2bed052a6b125aafe3365" Jul 03 14:39:35.774206 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.774141 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:39:35.955029 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.954996 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istiod-ca-cert\") pod \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " Jul 03 14:39:35.955213 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.955055 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-token\") pod \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " Jul 03 14:39:35.955213 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.955081 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-workload-certs\") pod \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " Jul 03 14:39:35.955213 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.955129 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-workload-socket\") pod \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " Jul 03 14:39:35.955213 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.955150 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6p2h\" (UniqueName: \"kubernetes.io/projected/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-kube-api-access-p6p2h\") pod \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " Jul 03 14:39:35.955213 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.955177 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-credential-socket\") pod \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " Jul 03 14:39:35.955470 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.955213 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-envoy\") pod \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " Jul 03 14:39:35.955528 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.955488 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" (UID: "7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:39:35.955587 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.955495 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" (UID: "7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:39:35.955754 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.955576 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" (UID: "7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:39:35.955754 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.955607 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-podinfo\") pod \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " Jul 03 14:39:35.955754 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.955644 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-data\") pod \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\" (UID: \"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84\") " Jul 03 14:39:35.956033 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.956001 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-workload-socket\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:39:35.956033 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.956031 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-credential-socket\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:39:35.956166 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.956046 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-workload-certs\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:39:35.956228 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.956204 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" (UID: "7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:39:35.956560 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.956539 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" (UID: "7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:39:35.957543 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.957518 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" (UID: "7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:39:35.957844 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.957818 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" (UID: "7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 03 14:39:35.958044 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.958010 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-kube-api-access-p6p2h" (OuterVolumeSpecName: "kube-api-access-p6p2h") pod "7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" (UID: "7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84"). InnerVolumeSpecName "kube-api-access-p6p2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:39:35.958044 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:35.958019 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" (UID: "7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:39:36.056849 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:36.056811 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6p2h\" (UniqueName: \"kubernetes.io/projected/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-kube-api-access-p6p2h\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:39:36.056849 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:36.056846 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-envoy\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:39:36.056849 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:36.056856 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-podinfo\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:39:36.057092 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:36.056864 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-data\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:39:36.057092 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:36.056872 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istiod-ca-cert\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:39:36.057092 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:36.056879 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84-istio-token\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:39:36.665802 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:36.665773 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" Jul 03 14:39:36.665802 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:36.665780 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47" event={"ID":"7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84","Type":"ContainerDied","Data":"c0b53103a26ff966cf78391431aba1b2d4c1802e61bfa04aeffb73cdbb5f0bca"} Jul 03 14:39:36.666285 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:36.665847 2576 scope.go:117] "RemoveContainer" containerID="cd9b1e6396fd3967a639b6693b6cececcca45b9d8023e204c8dc76cc35a70177" Jul 03 14:39:36.703540 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:36.703506 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47"] Jul 03 14:39:36.708032 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:36.707996 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-072d4360-openshift-default-6749ff5f79-b7c47"] Jul 03 14:39:38.167967 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:39:38.167933 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" path="/var/lib/kubelet/pods/7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84/volumes" Jul 03 14:40:25.334196 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.334164 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh"] Jul 03 14:40:25.334813 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.334636 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f534a30-6a94-4dda-9b9c-83e00c232b87" containerName="maas-api" Jul 03 14:40:25.334813 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.334683 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f534a30-6a94-4dda-9b9c-83e00c232b87" containerName="maas-api" Jul 03 14:40:25.334813 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.334696 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" containerName="istio-proxy" Jul 03 14:40:25.334813 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.334706 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" containerName="istio-proxy" Jul 03 14:40:25.334813 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.334716 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" containerName="istio-proxy" Jul 03 14:40:25.334813 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.334724 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" containerName="istio-proxy" Jul 03 14:40:25.335136 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.334873 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f534a30-6a94-4dda-9b9c-83e00c232b87" containerName="maas-api" Jul 03 14:40:25.335136 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.334889 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" containerName="istio-proxy" Jul 03 14:40:25.335136 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.334919 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c03b8d0-af95-4c62-8b9e-6dcd8b08aa84" containerName="istio-proxy" Jul 03 14:40:25.338046 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.338027 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.341492 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.341474 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2d96f7-openshift-default-dockercfg-wnlbd\"" Jul 03 14:40:25.352134 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.352110 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh"] Jul 03 14:40:25.404118 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.404083 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-workload-certs\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.404333 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.404133 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-envoy\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.404333 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.404192 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-token\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.404333 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.404253 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istiod-ca-cert\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.404333 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.404283 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-credential-socket\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.404333 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.404332 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-workload-socket\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.404542 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.404372 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-podinfo\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.404542 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.404402 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68vfv\" (UniqueName: \"kubernetes.io/projected/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-kube-api-access-68vfv\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.404542 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.404458 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-data\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.505364 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.505269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-data\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.505364 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.505331 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-workload-certs\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.505623 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.505371 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-envoy\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.505623 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.505402 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-token\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.505623 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.505443 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istiod-ca-cert\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.505623 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.505472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-credential-socket\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.505623 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.505520 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-workload-socket\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.505623 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.505545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-podinfo\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.505623 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.505576 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vfv\" (UniqueName: \"kubernetes.io/projected/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-kube-api-access-68vfv\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.506028 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.505746 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-data\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.506089 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.506047 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-workload-certs\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.506145 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.506112 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-credential-socket\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.506320 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.506300 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-workload-socket\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.506379 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.506319 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istiod-ca-cert\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.507880 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.507860 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-envoy\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.508688 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.508623 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-podinfo\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.516890 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.516864 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68vfv\" (UniqueName: \"kubernetes.io/projected/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-kube-api-access-68vfv\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.517843 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.517827 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-token\") pod \"e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.648577 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.648543 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:25.792397 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.792367 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh"] Jul 03 14:40:25.794233 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:40:25.794208 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe19bde6_1adf_4e94_8cb7_ac00981af5ff.slice/crio-d146d9f2a7f14e347598fc258f868fa80b57fef1bf1261b1620da9c9b702e901 WatchSource:0}: Error finding container d146d9f2a7f14e347598fc258f868fa80b57fef1bf1261b1620da9c9b702e901: Status 404 returned error can't find the container with id d146d9f2a7f14e347598fc258f868fa80b57fef1bf1261b1620da9c9b702e901 Jul 03 14:40:25.796301 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.796266 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:40:25.796398 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.796335 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:40:25.796398 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.796367 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:40:25.895108 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.895066 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" event={"ID":"fe19bde6-1adf-4e94-8cb7-ac00981af5ff","Type":"ContainerStarted","Data":"744da348bacd5546a96352b1392ac03a1fdc3a138a74aa021d70d636348dd773"} Jul 03 14:40:25.895303 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.895175 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" event={"ID":"fe19bde6-1adf-4e94-8cb7-ac00981af5ff","Type":"ContainerStarted","Data":"d146d9f2a7f14e347598fc258f868fa80b57fef1bf1261b1620da9c9b702e901"} Jul 03 14:40:25.917151 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:25.917098 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" podStartSLOduration=0.917082225 podStartE2EDuration="917.082225ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:40:25.915002724 +0000 UTC m=+1904.271287377" watchObservedRunningTime="2026-07-03 14:40:25.917082225 +0000 UTC m=+1904.273366845" Jul 03 14:40:26.649401 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:26.649368 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:26.654171 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:26.654149 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:26.900092 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:26.900001 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:26.901037 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:26.901019 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:28.187586 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:28.187555 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb"] Jul 03 14:40:28.191053 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:28.191032 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" Jul 03 14:40:28.194832 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:28.194802 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-2d96f7-serving-cert\"" Jul 03 14:40:28.201827 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:28.201791 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb"] Jul 03 14:40:28.224767 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:28.224730 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f200caf-e35c-4c21-ad8f-71f295401e6e-maas-api-tls\") pod \"maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb\" (UID: \"9f200caf-e35c-4c21-ad8f-71f295401e6e\") " pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" Jul 03 14:40:28.224949 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:28.224784 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f200caf-e35c-4c21-ad8f-71f295401e6e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb\" (UID: \"9f200caf-e35c-4c21-ad8f-71f295401e6e\") " pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" Jul 03 14:40:28.224949 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:28.224839 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sknv\" (UniqueName: \"kubernetes.io/projected/9f200caf-e35c-4c21-ad8f-71f295401e6e-kube-api-access-9sknv\") pod \"maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb\" (UID: \"9f200caf-e35c-4c21-ad8f-71f295401e6e\") " pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" Jul 03 14:40:28.325587 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:28.325549 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sknv\" (UniqueName: \"kubernetes.io/projected/9f200caf-e35c-4c21-ad8f-71f295401e6e-kube-api-access-9sknv\") pod \"maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb\" (UID: \"9f200caf-e35c-4c21-ad8f-71f295401e6e\") " pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" Jul 03 14:40:28.325801 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:28.325636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f200caf-e35c-4c21-ad8f-71f295401e6e-maas-api-tls\") pod \"maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb\" (UID: \"9f200caf-e35c-4c21-ad8f-71f295401e6e\") " pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" Jul 03 14:40:28.325801 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:28.325689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f200caf-e35c-4c21-ad8f-71f295401e6e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb\" (UID: \"9f200caf-e35c-4c21-ad8f-71f295401e6e\") " pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" Jul 03 14:40:28.325881 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:40:28.325798 2576 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-2d96f7-serving-cert: secret "maas-api-e2e-inf-a-2d96f7-serving-cert" not found Jul 03 14:40:28.325916 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:40:28.325880 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f200caf-e35c-4c21-ad8f-71f295401e6e-maas-api-tls podName:9f200caf-e35c-4c21-ad8f-71f295401e6e nodeName:}" failed. No retries permitted until 2026-07-03 14:40:28.825856733 +0000 UTC m=+1907.182141348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f200caf-e35c-4c21-ad8f-71f295401e6e-maas-api-tls") pod "maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" (UID: "9f200caf-e35c-4c21-ad8f-71f295401e6e") : secret "maas-api-e2e-inf-a-2d96f7-serving-cert" not found Jul 03 14:40:28.326280 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:28.326261 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f200caf-e35c-4c21-ad8f-71f295401e6e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb\" (UID: \"9f200caf-e35c-4c21-ad8f-71f295401e6e\") " pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" Jul 03 14:40:28.335945 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:28.335913 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sknv\" (UniqueName: \"kubernetes.io/projected/9f200caf-e35c-4c21-ad8f-71f295401e6e-kube-api-access-9sknv\") pod \"maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb\" (UID: \"9f200caf-e35c-4c21-ad8f-71f295401e6e\") " pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" Jul 03 14:40:28.830005 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:28.829964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f200caf-e35c-4c21-ad8f-71f295401e6e-maas-api-tls\") pod \"maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb\" (UID: \"9f200caf-e35c-4c21-ad8f-71f295401e6e\") " pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" Jul 03 14:40:28.832527 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:28.832505 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f200caf-e35c-4c21-ad8f-71f295401e6e-maas-api-tls\") pod \"maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb\" (UID: \"9f200caf-e35c-4c21-ad8f-71f295401e6e\") " pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" Jul 03 14:40:29.104730 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:29.104613 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" Jul 03 14:40:29.244988 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:29.244955 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb"] Jul 03 14:40:29.249011 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:40:29.248971 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f200caf_e35c_4c21_ad8f_71f295401e6e.slice/crio-09b980002e306f7bf25ae53ef3c220e9425e63c4fadb8ae5a9e67a3e858dd668 WatchSource:0}: Error finding container 09b980002e306f7bf25ae53ef3c220e9425e63c4fadb8ae5a9e67a3e858dd668: Status 404 returned error can't find the container with id 09b980002e306f7bf25ae53ef3c220e9425e63c4fadb8ae5a9e67a3e858dd668 Jul 03 14:40:29.250743 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:29.250722 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 03 14:40:29.916242 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:29.916201 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" event={"ID":"9f200caf-e35c-4c21-ad8f-71f295401e6e","Type":"ContainerStarted","Data":"285be894f988d93548cdcc96003b9ac3e440a9a6fffa97c28f427ed7ad9a6f68"} Jul 03 14:40:29.916421 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:29.916259 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" event={"ID":"9f200caf-e35c-4c21-ad8f-71f295401e6e","Type":"ContainerStarted","Data":"09b980002e306f7bf25ae53ef3c220e9425e63c4fadb8ae5a9e67a3e858dd668"} Jul 03 14:40:29.916421 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:29.916298 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" Jul 03 14:40:29.935540 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:29.935480 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" podStartSLOduration=1.3888403839999999 podStartE2EDuration="1.935461979s" totalImagesPullingTime="546.621595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:40:29.934771529 +0000 UTC m=+1908.291056158" watchObservedRunningTime="2026-07-03 14:40:29.935461979 +0000 UTC m=+1908.291746599" Jul 03 14:40:35.929530 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:35.929487 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" Jul 03 14:40:38.708709 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:38.708650 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv"] Jul 03 14:40:38.713520 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:38.713494 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" Jul 03 14:40:38.719072 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:38.718996 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-7195a2-serving-cert\"" Jul 03 14:40:38.729930 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:38.729900 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv"] Jul 03 14:40:38.818791 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:38.818749 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd07ea1-95c4-4390-8e70-9cccd6cab486-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv\" (UID: \"6cd07ea1-95c4-4390-8e70-9cccd6cab486\") " pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" Jul 03 14:40:38.818983 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:38.818824 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cd07ea1-95c4-4390-8e70-9cccd6cab486-maas-api-tls\") pod \"maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv\" (UID: \"6cd07ea1-95c4-4390-8e70-9cccd6cab486\") " pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" Jul 03 14:40:38.818983 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:38.818866 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhdc2\" (UniqueName: \"kubernetes.io/projected/6cd07ea1-95c4-4390-8e70-9cccd6cab486-kube-api-access-hhdc2\") pod \"maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv\" (UID: \"6cd07ea1-95c4-4390-8e70-9cccd6cab486\") " pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" Jul 03 14:40:38.920144 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:38.920089 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd07ea1-95c4-4390-8e70-9cccd6cab486-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv\" (UID: \"6cd07ea1-95c4-4390-8e70-9cccd6cab486\") " pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" Jul 03 14:40:38.920352 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:38.920175 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cd07ea1-95c4-4390-8e70-9cccd6cab486-maas-api-tls\") pod \"maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv\" (UID: \"6cd07ea1-95c4-4390-8e70-9cccd6cab486\") " pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" Jul 03 14:40:38.920352 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:38.920223 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhdc2\" (UniqueName: \"kubernetes.io/projected/6cd07ea1-95c4-4390-8e70-9cccd6cab486-kube-api-access-hhdc2\") pod \"maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv\" (UID: \"6cd07ea1-95c4-4390-8e70-9cccd6cab486\") " pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" Jul 03 14:40:38.920352 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:40:38.920342 2576 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-7195a2-serving-cert: secret "maas-api-e2e-inf-b-7195a2-serving-cert" not found Jul 03 14:40:38.920533 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:40:38.920439 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cd07ea1-95c4-4390-8e70-9cccd6cab486-maas-api-tls podName:6cd07ea1-95c4-4390-8e70-9cccd6cab486 nodeName:}" failed. No retries permitted until 2026-07-03 14:40:39.420414064 +0000 UTC m=+1917.776698691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6cd07ea1-95c4-4390-8e70-9cccd6cab486-maas-api-tls") pod "maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" (UID: "6cd07ea1-95c4-4390-8e70-9cccd6cab486") : secret "maas-api-e2e-inf-b-7195a2-serving-cert" not found Jul 03 14:40:38.921062 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:38.921037 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd07ea1-95c4-4390-8e70-9cccd6cab486-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv\" (UID: \"6cd07ea1-95c4-4390-8e70-9cccd6cab486\") " pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" Jul 03 14:40:38.935805 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:38.935775 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhdc2\" (UniqueName: \"kubernetes.io/projected/6cd07ea1-95c4-4390-8e70-9cccd6cab486-kube-api-access-hhdc2\") pod \"maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv\" (UID: \"6cd07ea1-95c4-4390-8e70-9cccd6cab486\") " pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" Jul 03 14:40:39.425694 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:39.425642 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cd07ea1-95c4-4390-8e70-9cccd6cab486-maas-api-tls\") pod \"maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv\" (UID: \"6cd07ea1-95c4-4390-8e70-9cccd6cab486\") " pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" Jul 03 14:40:39.428294 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:39.428269 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cd07ea1-95c4-4390-8e70-9cccd6cab486-maas-api-tls\") pod \"maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv\" (UID: \"6cd07ea1-95c4-4390-8e70-9cccd6cab486\") " pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" Jul 03 14:40:39.628049 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:39.628008 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" Jul 03 14:40:39.785082 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:39.785053 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv"] Jul 03 14:40:39.787777 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:40:39.787744 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd07ea1_95c4_4390_8e70_9cccd6cab486.slice/crio-108fc99d6ece30470cc246d8ac6bd7cc47e1dc5cd102b2645e0e1a712700e41e WatchSource:0}: Error finding container 108fc99d6ece30470cc246d8ac6bd7cc47e1dc5cd102b2645e0e1a712700e41e: Status 404 returned error can't find the container with id 108fc99d6ece30470cc246d8ac6bd7cc47e1dc5cd102b2645e0e1a712700e41e Jul 03 14:40:39.971568 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:39.971479 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" event={"ID":"6cd07ea1-95c4-4390-8e70-9cccd6cab486","Type":"ContainerStarted","Data":"108fc99d6ece30470cc246d8ac6bd7cc47e1dc5cd102b2645e0e1a712700e41e"} Jul 03 14:40:40.979919 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:40.979107 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" event={"ID":"6cd07ea1-95c4-4390-8e70-9cccd6cab486","Type":"ContainerStarted","Data":"7a6c11e4efbd506827b652ece35eac671a2d339594f5962691bf4b848d412a01"} Jul 03 14:40:40.979919 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:40.979231 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" Jul 03 14:40:41.017764 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:41.017711 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" podStartSLOduration=2.673822141 podStartE2EDuration="3.017692697s" totalImagesPullingTime="343.870556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:40:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:40:41.016460892 +0000 UTC m=+1919.372745511" watchObservedRunningTime="2026-07-03 14:40:41.017692697 +0000 UTC m=+1919.373977317" Jul 03 14:40:41.902083 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:41.902044 2576 patch_prober.go:28] interesting pod/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 03 14:40:41.902287 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:41.902110 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" podUID="fe19bde6-1adf-4e94-8cb7-ac00981af5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 03 14:40:41.985014 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:41.984978 2576 generic.go:356] "Generic (PLEG): container finished" podID="fe19bde6-1adf-4e94-8cb7-ac00981af5ff" containerID="744da348bacd5546a96352b1392ac03a1fdc3a138a74aa021d70d636348dd773" exitCode=0 Jul 03 14:40:41.985402 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:41.985049 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" event={"ID":"fe19bde6-1adf-4e94-8cb7-ac00981af5ff","Type":"ContainerDied","Data":"744da348bacd5546a96352b1392ac03a1fdc3a138a74aa021d70d636348dd773"} Jul 03 14:40:41.985524 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:41.985507 2576 scope.go:117] "RemoveContainer" containerID="744da348bacd5546a96352b1392ac03a1fdc3a138a74aa021d70d636348dd773" Jul 03 14:40:41.986543 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:41.986509 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:40:41.986602 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:41.986582 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:40:41.986638 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:41.986615 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:40:42.992187 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:42.992147 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" event={"ID":"fe19bde6-1adf-4e94-8cb7-ac00981af5ff","Type":"ContainerStarted","Data":"c972cbb99a058431c7aac81ae532aedd1ad3e3ba35404888d56e5c4d4c1b7d81"} Jul 03 14:40:43.649625 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:43.649582 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:43.649865 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:43.649841 2576 patch_prober.go:28] interesting pod/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 03 14:40:43.649911 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:43.649890 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" podUID="fe19bde6-1adf-4e94-8cb7-ac00981af5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 03 14:40:44.649233 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:44.649192 2576 patch_prober.go:28] interesting pod/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 03 14:40:44.649623 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:44.649258 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" podUID="fe19bde6-1adf-4e94-8cb7-ac00981af5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 03 14:40:45.649461 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:45.649419 2576 patch_prober.go:28] interesting pod/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 03 14:40:45.649977 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:45.649484 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" podUID="fe19bde6-1adf-4e94-8cb7-ac00981af5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 03 14:40:46.650035 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:46.649997 2576 patch_prober.go:28] interesting pod/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 03 14:40:46.650425 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:46.650058 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" podUID="fe19bde6-1adf-4e94-8cb7-ac00981af5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 03 14:40:46.990127 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:46.990049 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" Jul 03 14:40:47.652815 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:47.652777 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:47.653252 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:47.653186 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:40:47.653921 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:40:47.653906 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:42:30.093813 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.093775 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb"] Jul 03 14:42:30.094344 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.094039 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" podUID="9f200caf-e35c-4c21-ad8f-71f295401e6e" containerName="maas-api" containerID="cri-o://285be894f988d93548cdcc96003b9ac3e440a9a6fffa97c28f427ed7ad9a6f68" gracePeriod=30 Jul 03 14:42:30.354807 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.354744 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" Jul 03 14:42:30.443593 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.443557 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f200caf-e35c-4c21-ad8f-71f295401e6e-openshift-service-ca\") pod \"9f200caf-e35c-4c21-ad8f-71f295401e6e\" (UID: \"9f200caf-e35c-4c21-ad8f-71f295401e6e\") " Jul 03 14:42:30.443811 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.443619 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f200caf-e35c-4c21-ad8f-71f295401e6e-maas-api-tls\") pod \"9f200caf-e35c-4c21-ad8f-71f295401e6e\" (UID: \"9f200caf-e35c-4c21-ad8f-71f295401e6e\") " Jul 03 14:42:30.443811 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.443725 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sknv\" (UniqueName: \"kubernetes.io/projected/9f200caf-e35c-4c21-ad8f-71f295401e6e-kube-api-access-9sknv\") pod \"9f200caf-e35c-4c21-ad8f-71f295401e6e\" (UID: \"9f200caf-e35c-4c21-ad8f-71f295401e6e\") " Jul 03 14:42:30.443943 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.443842 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f200caf-e35c-4c21-ad8f-71f295401e6e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f200caf-e35c-4c21-ad8f-71f295401e6e" (UID: "9f200caf-e35c-4c21-ad8f-71f295401e6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:42:30.443999 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.443979 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f200caf-e35c-4c21-ad8f-71f295401e6e-openshift-service-ca\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:42:30.445943 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.445910 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f200caf-e35c-4c21-ad8f-71f295401e6e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f200caf-e35c-4c21-ad8f-71f295401e6e" (UID: "9f200caf-e35c-4c21-ad8f-71f295401e6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 14:42:30.446057 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.445955 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f200caf-e35c-4c21-ad8f-71f295401e6e-kube-api-access-9sknv" (OuterVolumeSpecName: "kube-api-access-9sknv") pod "9f200caf-e35c-4c21-ad8f-71f295401e6e" (UID: "9f200caf-e35c-4c21-ad8f-71f295401e6e"). InnerVolumeSpecName "kube-api-access-9sknv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:42:30.507104 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.507064 2576 generic.go:356] "Generic (PLEG): container finished" podID="9f200caf-e35c-4c21-ad8f-71f295401e6e" containerID="285be894f988d93548cdcc96003b9ac3e440a9a6fffa97c28f427ed7ad9a6f68" exitCode=0 Jul 03 14:42:30.507275 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.507115 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" Jul 03 14:42:30.507275 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.507151 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" event={"ID":"9f200caf-e35c-4c21-ad8f-71f295401e6e","Type":"ContainerDied","Data":"285be894f988d93548cdcc96003b9ac3e440a9a6fffa97c28f427ed7ad9a6f68"} Jul 03 14:42:30.507275 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.507207 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb" event={"ID":"9f200caf-e35c-4c21-ad8f-71f295401e6e","Type":"ContainerDied","Data":"09b980002e306f7bf25ae53ef3c220e9425e63c4fadb8ae5a9e67a3e858dd668"} Jul 03 14:42:30.507275 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.507231 2576 scope.go:117] "RemoveContainer" containerID="285be894f988d93548cdcc96003b9ac3e440a9a6fffa97c28f427ed7ad9a6f68" Jul 03 14:42:30.516872 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.516852 2576 scope.go:117] "RemoveContainer" containerID="285be894f988d93548cdcc96003b9ac3e440a9a6fffa97c28f427ed7ad9a6f68" Jul 03 14:42:30.517140 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:42:30.517122 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"285be894f988d93548cdcc96003b9ac3e440a9a6fffa97c28f427ed7ad9a6f68\": container with ID starting with 285be894f988d93548cdcc96003b9ac3e440a9a6fffa97c28f427ed7ad9a6f68 not found: ID does not exist" containerID="285be894f988d93548cdcc96003b9ac3e440a9a6fffa97c28f427ed7ad9a6f68" Jul 03 14:42:30.517184 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.517149 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285be894f988d93548cdcc96003b9ac3e440a9a6fffa97c28f427ed7ad9a6f68"} err="failed to get container status \"285be894f988d93548cdcc96003b9ac3e440a9a6fffa97c28f427ed7ad9a6f68\": rpc error: code = NotFound desc = could not find container \"285be894f988d93548cdcc96003b9ac3e440a9a6fffa97c28f427ed7ad9a6f68\": container with ID starting with 285be894f988d93548cdcc96003b9ac3e440a9a6fffa97c28f427ed7ad9a6f68 not found: ID does not exist" Jul 03 14:42:30.529796 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.529765 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb"] Jul 03 14:42:30.534113 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.534089 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-2d96f7-6c6947c9f8-npmnb"] Jul 03 14:42:30.544537 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.544500 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sknv\" (UniqueName: \"kubernetes.io/projected/9f200caf-e35c-4c21-ad8f-71f295401e6e-kube-api-access-9sknv\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:42:30.544537 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:30.544538 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f200caf-e35c-4c21-ad8f-71f295401e6e-maas-api-tls\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:42:32.167868 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:42:32.167838 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f200caf-e35c-4c21-ad8f-71f295401e6e" path="/var/lib/kubelet/pods/9f200caf-e35c-4c21-ad8f-71f295401e6e/volumes" Jul 03 14:43:12.094280 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:12.094192 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh"] Jul 03 14:43:12.094904 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:12.094504 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" podUID="fe19bde6-1adf-4e94-8cb7-ac00981af5ff" containerName="istio-proxy" containerID="cri-o://c972cbb99a058431c7aac81ae532aedd1ad3e3ba35404888d56e5c4d4c1b7d81" gracePeriod=30 Jul 03 14:43:13.165469 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:13.165432 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv"] Jul 03 14:43:13.165976 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:13.165709 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" podUID="6cd07ea1-95c4-4390-8e70-9cccd6cab486" containerName="maas-api" containerID="cri-o://7a6c11e4efbd506827b652ece35eac671a2d339594f5962691bf4b848d412a01" gracePeriod=30 Jul 03 14:43:14.413391 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:14.413365 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" Jul 03 14:43:14.503256 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:14.503179 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cd07ea1-95c4-4390-8e70-9cccd6cab486-maas-api-tls\") pod \"6cd07ea1-95c4-4390-8e70-9cccd6cab486\" (UID: \"6cd07ea1-95c4-4390-8e70-9cccd6cab486\") " Jul 03 14:43:14.503404 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:14.503287 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd07ea1-95c4-4390-8e70-9cccd6cab486-openshift-service-ca\") pod \"6cd07ea1-95c4-4390-8e70-9cccd6cab486\" (UID: \"6cd07ea1-95c4-4390-8e70-9cccd6cab486\") " Jul 03 14:43:14.503404 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:14.503315 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhdc2\" (UniqueName: \"kubernetes.io/projected/6cd07ea1-95c4-4390-8e70-9cccd6cab486-kube-api-access-hhdc2\") pod \"6cd07ea1-95c4-4390-8e70-9cccd6cab486\" (UID: \"6cd07ea1-95c4-4390-8e70-9cccd6cab486\") " Jul 03 14:43:14.503711 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:14.503648 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd07ea1-95c4-4390-8e70-9cccd6cab486-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6cd07ea1-95c4-4390-8e70-9cccd6cab486" (UID: "6cd07ea1-95c4-4390-8e70-9cccd6cab486"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:43:14.505364 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:14.505344 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd07ea1-95c4-4390-8e70-9cccd6cab486-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6cd07ea1-95c4-4390-8e70-9cccd6cab486" (UID: "6cd07ea1-95c4-4390-8e70-9cccd6cab486"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 14:43:14.505538 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:14.505516 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd07ea1-95c4-4390-8e70-9cccd6cab486-kube-api-access-hhdc2" (OuterVolumeSpecName: "kube-api-access-hhdc2") pod "6cd07ea1-95c4-4390-8e70-9cccd6cab486" (UID: "6cd07ea1-95c4-4390-8e70-9cccd6cab486"). InnerVolumeSpecName "kube-api-access-hhdc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:43:14.604051 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:14.604016 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd07ea1-95c4-4390-8e70-9cccd6cab486-openshift-service-ca\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:14.604051 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:14.604044 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhdc2\" (UniqueName: \"kubernetes.io/projected/6cd07ea1-95c4-4390-8e70-9cccd6cab486-kube-api-access-hhdc2\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:14.604051 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:14.604055 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cd07ea1-95c4-4390-8e70-9cccd6cab486-maas-api-tls\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:14.712025 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:14.711992 2576 generic.go:356] "Generic (PLEG): container finished" podID="6cd07ea1-95c4-4390-8e70-9cccd6cab486" containerID="7a6c11e4efbd506827b652ece35eac671a2d339594f5962691bf4b848d412a01" exitCode=0 Jul 03 14:43:14.712201 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:14.712041 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" Jul 03 14:43:14.712201 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:14.712072 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" event={"ID":"6cd07ea1-95c4-4390-8e70-9cccd6cab486","Type":"ContainerDied","Data":"7a6c11e4efbd506827b652ece35eac671a2d339594f5962691bf4b848d412a01"} Jul 03 14:43:14.712201 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:14.712111 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv" event={"ID":"6cd07ea1-95c4-4390-8e70-9cccd6cab486","Type":"ContainerDied","Data":"108fc99d6ece30470cc246d8ac6bd7cc47e1dc5cd102b2645e0e1a712700e41e"} Jul 03 14:43:14.712201 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:14.712132 2576 scope.go:117] "RemoveContainer" containerID="7a6c11e4efbd506827b652ece35eac671a2d339594f5962691bf4b848d412a01" Jul 03 14:43:14.721973 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:14.721955 2576 scope.go:117] "RemoveContainer" containerID="7a6c11e4efbd506827b652ece35eac671a2d339594f5962691bf4b848d412a01" Jul 03 14:43:14.722249 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:43:14.722230 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a6c11e4efbd506827b652ece35eac671a2d339594f5962691bf4b848d412a01\": container with ID starting with 7a6c11e4efbd506827b652ece35eac671a2d339594f5962691bf4b848d412a01 not found: ID does not exist" containerID="7a6c11e4efbd506827b652ece35eac671a2d339594f5962691bf4b848d412a01" Jul 03 14:43:14.722319 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:14.722262 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6c11e4efbd506827b652ece35eac671a2d339594f5962691bf4b848d412a01"} err="failed to get container status \"7a6c11e4efbd506827b652ece35eac671a2d339594f5962691bf4b848d412a01\": rpc error: code = NotFound desc = could not find container \"7a6c11e4efbd506827b652ece35eac671a2d339594f5962691bf4b848d412a01\": container with ID starting with 7a6c11e4efbd506827b652ece35eac671a2d339594f5962691bf4b848d412a01 not found: ID does not exist" Jul 03 14:43:14.734737 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:14.734701 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv"] Jul 03 14:43:14.740527 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:14.740504 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-7195a2-787f6b9f54-tsmjv"] Jul 03 14:43:16.167873 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:16.167839 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cd07ea1-95c4-4390-8e70-9cccd6cab486" path="/var/lib/kubelet/pods/6cd07ea1-95c4-4390-8e70-9cccd6cab486/volumes" Jul 03 14:43:17.360019 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.359952 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:43:17.432511 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.432473 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68vfv\" (UniqueName: \"kubernetes.io/projected/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-kube-api-access-68vfv\") pod \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " Jul 03 14:43:17.432511 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.432517 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-podinfo\") pod \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " Jul 03 14:43:17.432798 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.432553 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-credential-socket\") pod \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " Jul 03 14:43:17.432798 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.432576 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-token\") pod \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " Jul 03 14:43:17.432798 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.432624 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-workload-certs\") pod \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " Jul 03 14:43:17.432798 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.432743 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-envoy\") pod \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " Jul 03 14:43:17.432798 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.432792 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-workload-socket\") pod \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " Jul 03 14:43:17.433044 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.432826 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istiod-ca-cert\") pod \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " Jul 03 14:43:17.433044 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.432851 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-data\") pod \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\" (UID: \"fe19bde6-1adf-4e94-8cb7-ac00981af5ff\") " Jul 03 14:43:17.433044 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.432868 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fe19bde6-1adf-4e94-8cb7-ac00981af5ff" (UID: "fe19bde6-1adf-4e94-8cb7-ac00981af5ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:43:17.433044 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.432953 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fe19bde6-1adf-4e94-8cb7-ac00981af5ff" (UID: "fe19bde6-1adf-4e94-8cb7-ac00981af5ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:43:17.433255 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.433063 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fe19bde6-1adf-4e94-8cb7-ac00981af5ff" (UID: "fe19bde6-1adf-4e94-8cb7-ac00981af5ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:43:17.433255 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.433134 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fe19bde6-1adf-4e94-8cb7-ac00981af5ff" (UID: "fe19bde6-1adf-4e94-8cb7-ac00981af5ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:43:17.433255 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.433235 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-workload-socket\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:17.433255 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.433254 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istiod-ca-cert\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:17.433450 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.433268 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-credential-socket\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:17.433450 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.433283 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-workload-certs\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:17.433855 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.433832 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-data" (OuterVolumeSpecName: "istio-data") pod "fe19bde6-1adf-4e94-8cb7-ac00981af5ff" (UID: "fe19bde6-1adf-4e94-8cb7-ac00981af5ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:43:17.435026 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.434999 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-kube-api-access-68vfv" (OuterVolumeSpecName: "kube-api-access-68vfv") pod "fe19bde6-1adf-4e94-8cb7-ac00981af5ff" (UID: "fe19bde6-1adf-4e94-8cb7-ac00981af5ff"). InnerVolumeSpecName "kube-api-access-68vfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:43:17.435145 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.435038 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fe19bde6-1adf-4e94-8cb7-ac00981af5ff" (UID: "fe19bde6-1adf-4e94-8cb7-ac00981af5ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:43:17.435364 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.435348 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-token" (OuterVolumeSpecName: "istio-token") pod "fe19bde6-1adf-4e94-8cb7-ac00981af5ff" (UID: "fe19bde6-1adf-4e94-8cb7-ac00981af5ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:43:17.435425 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.435388 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fe19bde6-1adf-4e94-8cb7-ac00981af5ff" (UID: "fe19bde6-1adf-4e94-8cb7-ac00981af5ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 03 14:43:17.534406 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.534371 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-envoy\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:17.534406 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.534399 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-data\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:17.534406 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.534410 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68vfv\" (UniqueName: \"kubernetes.io/projected/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-kube-api-access-68vfv\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:17.534646 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.534420 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-podinfo\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:17.534646 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.534428 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe19bde6-1adf-4e94-8cb7-ac00981af5ff-istio-token\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:17.729252 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.729161 2576 generic.go:356] "Generic (PLEG): container finished" podID="fe19bde6-1adf-4e94-8cb7-ac00981af5ff" containerID="c972cbb99a058431c7aac81ae532aedd1ad3e3ba35404888d56e5c4d4c1b7d81" exitCode=0 Jul 03 14:43:17.729252 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.729209 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" Jul 03 14:43:17.729252 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.729227 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" event={"ID":"fe19bde6-1adf-4e94-8cb7-ac00981af5ff","Type":"ContainerDied","Data":"c972cbb99a058431c7aac81ae532aedd1ad3e3ba35404888d56e5c4d4c1b7d81"} Jul 03 14:43:17.729457 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.729270 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh" event={"ID":"fe19bde6-1adf-4e94-8cb7-ac00981af5ff","Type":"ContainerDied","Data":"d146d9f2a7f14e347598fc258f868fa80b57fef1bf1261b1620da9c9b702e901"} Jul 03 14:43:17.729457 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.729307 2576 scope.go:117] "RemoveContainer" containerID="c972cbb99a058431c7aac81ae532aedd1ad3e3ba35404888d56e5c4d4c1b7d81" Jul 03 14:43:17.738906 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.738888 2576 scope.go:117] "RemoveContainer" containerID="744da348bacd5546a96352b1392ac03a1fdc3a138a74aa021d70d636348dd773" Jul 03 14:43:17.746728 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.746689 2576 scope.go:117] "RemoveContainer" containerID="c972cbb99a058431c7aac81ae532aedd1ad3e3ba35404888d56e5c4d4c1b7d81" Jul 03 14:43:17.746937 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:43:17.746919 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c972cbb99a058431c7aac81ae532aedd1ad3e3ba35404888d56e5c4d4c1b7d81\": container with ID starting with c972cbb99a058431c7aac81ae532aedd1ad3e3ba35404888d56e5c4d4c1b7d81 not found: ID does not exist" containerID="c972cbb99a058431c7aac81ae532aedd1ad3e3ba35404888d56e5c4d4c1b7d81" Jul 03 14:43:17.746982 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.746947 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c972cbb99a058431c7aac81ae532aedd1ad3e3ba35404888d56e5c4d4c1b7d81"} err="failed to get container status \"c972cbb99a058431c7aac81ae532aedd1ad3e3ba35404888d56e5c4d4c1b7d81\": rpc error: code = NotFound desc = could not find container \"c972cbb99a058431c7aac81ae532aedd1ad3e3ba35404888d56e5c4d4c1b7d81\": container with ID starting with c972cbb99a058431c7aac81ae532aedd1ad3e3ba35404888d56e5c4d4c1b7d81 not found: ID does not exist" Jul 03 14:43:17.746982 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.746963 2576 scope.go:117] "RemoveContainer" containerID="744da348bacd5546a96352b1392ac03a1fdc3a138a74aa021d70d636348dd773" Jul 03 14:43:17.747173 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:43:17.747151 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744da348bacd5546a96352b1392ac03a1fdc3a138a74aa021d70d636348dd773\": container with ID starting with 744da348bacd5546a96352b1392ac03a1fdc3a138a74aa021d70d636348dd773 not found: ID does not exist" containerID="744da348bacd5546a96352b1392ac03a1fdc3a138a74aa021d70d636348dd773" Jul 03 14:43:17.747219 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.747181 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744da348bacd5546a96352b1392ac03a1fdc3a138a74aa021d70d636348dd773"} err="failed to get container status \"744da348bacd5546a96352b1392ac03a1fdc3a138a74aa021d70d636348dd773\": rpc error: code = NotFound desc = could not find container \"744da348bacd5546a96352b1392ac03a1fdc3a138a74aa021d70d636348dd773\": container with ID starting with 744da348bacd5546a96352b1392ac03a1fdc3a138a74aa021d70d636348dd773 not found: ID does not exist" Jul 03 14:43:17.751004 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.750982 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh"] Jul 03 14:43:17.754781 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:17.754759 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2d96f7-openshift-default-7956c6cc45-tcgrh"] Jul 03 14:43:18.167608 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:18.167573 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe19bde6-1adf-4e94-8cb7-ac00981af5ff" path="/var/lib/kubelet/pods/fe19bde6-1adf-4e94-8cb7-ac00981af5ff/volumes" Jul 03 14:43:20.159280 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.159239 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w"] Jul 03 14:43:20.159768 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.159747 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe19bde6-1adf-4e94-8cb7-ac00981af5ff" containerName="istio-proxy" Jul 03 14:43:20.159864 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.159771 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe19bde6-1adf-4e94-8cb7-ac00981af5ff" containerName="istio-proxy" Jul 03 14:43:20.159864 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.159785 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe19bde6-1adf-4e94-8cb7-ac00981af5ff" containerName="istio-proxy" Jul 03 14:43:20.159864 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.159794 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe19bde6-1adf-4e94-8cb7-ac00981af5ff" containerName="istio-proxy" Jul 03 14:43:20.159864 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.159805 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f200caf-e35c-4c21-ad8f-71f295401e6e" containerName="maas-api" Jul 03 14:43:20.159864 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.159814 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f200caf-e35c-4c21-ad8f-71f295401e6e" containerName="maas-api" Jul 03 14:43:20.159864 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.159835 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd07ea1-95c4-4390-8e70-9cccd6cab486" containerName="maas-api" Jul 03 14:43:20.159864 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.159843 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd07ea1-95c4-4390-8e70-9cccd6cab486" containerName="maas-api" Jul 03 14:43:20.160204 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.159932 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cd07ea1-95c4-4390-8e70-9cccd6cab486" containerName="maas-api" Jul 03 14:43:20.160204 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.159947 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe19bde6-1adf-4e94-8cb7-ac00981af5ff" containerName="istio-proxy" Jul 03 14:43:20.160204 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.159957 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe19bde6-1adf-4e94-8cb7-ac00981af5ff" containerName="istio-proxy" Jul 03 14:43:20.160204 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.159971 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f200caf-e35c-4c21-ad8f-71f295401e6e" containerName="maas-api" Jul 03 14:43:20.164537 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.164509 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.167499 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.167480 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4cf4ef-openshift-default-dockercfg-jmgt4\"" Jul 03 14:43:20.179999 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.179977 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w"] Jul 03 14:43:20.258561 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.258530 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-istio-envoy\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.258561 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.258568 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-credential-socket\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.258783 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.258590 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-workload-certs\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.258783 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.258640 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-workload-socket\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.258783 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.258694 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace7d172-70ea-470c-b659-73318430b0bd-istio-token\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.258783 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.258716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g58v\" (UniqueName: \"kubernetes.io/projected/ace7d172-70ea-470c-b659-73318430b0bd-kube-api-access-6g58v\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.258783 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.258749 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace7d172-70ea-470c-b659-73318430b0bd-istiod-ca-cert\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.258783 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.258775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace7d172-70ea-470c-b659-73318430b0bd-istio-podinfo\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.258976 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.258804 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-istio-data\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.360167 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.360135 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-istio-envoy\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.360353 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.360173 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-credential-socket\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.360353 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.360201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-workload-certs\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.360353 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.360222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-workload-socket\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.360353 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.360278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace7d172-70ea-470c-b659-73318430b0bd-istio-token\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.360353 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.360311 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g58v\" (UniqueName: \"kubernetes.io/projected/ace7d172-70ea-470c-b659-73318430b0bd-kube-api-access-6g58v\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.360353 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.360335 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace7d172-70ea-470c-b659-73318430b0bd-istiod-ca-cert\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.360743 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.360533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace7d172-70ea-470c-b659-73318430b0bd-istio-podinfo\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.360743 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.360604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-istio-data\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.360743 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.360613 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-workload-socket\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.360743 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.360724 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-credential-socket\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.360948 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.360777 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-workload-certs\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.361228 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.361201 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-istio-data\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.361722 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.361700 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace7d172-70ea-470c-b659-73318430b0bd-istiod-ca-cert\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.362779 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.362759 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-istio-envoy\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.363146 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.363130 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace7d172-70ea-470c-b659-73318430b0bd-istio-podinfo\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.371398 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.371377 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g58v\" (UniqueName: \"kubernetes.io/projected/ace7d172-70ea-470c-b659-73318430b0bd-kube-api-access-6g58v\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.376180 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.376149 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace7d172-70ea-470c-b659-73318430b0bd-istio-token\") pod \"e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.475935 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.475839 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:20.622459 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.622435 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w"] Jul 03 14:43:20.624495 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:43:20.624466 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace7d172_70ea_470c_b659_73318430b0bd.slice/crio-32b99e7859b02bf24ee6dbc0d62d13224e89b3c1b5e5c791bdb59bba6a28b375 WatchSource:0}: Error finding container 32b99e7859b02bf24ee6dbc0d62d13224e89b3c1b5e5c791bdb59bba6a28b375: Status 404 returned error can't find the container with id 32b99e7859b02bf24ee6dbc0d62d13224e89b3c1b5e5c791bdb59bba6a28b375 Jul 03 14:43:20.626529 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.626496 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:43:20.626602 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.626560 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:43:20.626602 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.626587 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:43:20.746449 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.746350 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" event={"ID":"ace7d172-70ea-470c-b659-73318430b0bd","Type":"ContainerStarted","Data":"8decb76e0ea6c76cb3ef93ea92edca012e58d47596692ffa99e0046d124b5a65"} Jul 03 14:43:20.746449 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.746423 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" event={"ID":"ace7d172-70ea-470c-b659-73318430b0bd","Type":"ContainerStarted","Data":"32b99e7859b02bf24ee6dbc0d62d13224e89b3c1b5e5c791bdb59bba6a28b375"} Jul 03 14:43:20.768459 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:20.768405 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" podStartSLOduration=0.768384415 podStartE2EDuration="768.384415ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:43:20.766991346 +0000 UTC m=+2079.123275967" watchObservedRunningTime="2026-07-03 14:43:20.768384415 +0000 UTC m=+2079.124669036" Jul 03 14:43:21.476526 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:21.476497 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:21.481237 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:21.481210 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:21.750877 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:21.750790 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:21.751839 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:21.751821 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:22.844411 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:22.844375 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv"] Jul 03 14:43:22.848554 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:22.848532 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" Jul 03 14:43:22.851407 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:22.851386 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-4cf4ef-serving-cert\"" Jul 03 14:43:22.859764 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:22.859740 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv"] Jul 03 14:43:22.885788 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:22.885752 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbb4a096-04ee-4d64-a99b-3930818313fb-maas-api-tls\") pod \"maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv\" (UID: \"bbb4a096-04ee-4d64-a99b-3930818313fb\") " pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" Jul 03 14:43:22.885788 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:22.885791 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxmpv\" (UniqueName: \"kubernetes.io/projected/bbb4a096-04ee-4d64-a99b-3930818313fb-kube-api-access-zxmpv\") pod \"maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv\" (UID: \"bbb4a096-04ee-4d64-a99b-3930818313fb\") " pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" Jul 03 14:43:22.886148 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:22.885996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb4a096-04ee-4d64-a99b-3930818313fb-openshift-service-ca\") pod \"maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv\" (UID: \"bbb4a096-04ee-4d64-a99b-3930818313fb\") " pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" Jul 03 14:43:22.986706 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:22.986643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb4a096-04ee-4d64-a99b-3930818313fb-openshift-service-ca\") pod \"maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv\" (UID: \"bbb4a096-04ee-4d64-a99b-3930818313fb\") " pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" Jul 03 14:43:22.986872 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:22.986740 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbb4a096-04ee-4d64-a99b-3930818313fb-maas-api-tls\") pod \"maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv\" (UID: \"bbb4a096-04ee-4d64-a99b-3930818313fb\") " pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" Jul 03 14:43:22.986872 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:22.986776 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxmpv\" (UniqueName: \"kubernetes.io/projected/bbb4a096-04ee-4d64-a99b-3930818313fb-kube-api-access-zxmpv\") pod \"maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv\" (UID: \"bbb4a096-04ee-4d64-a99b-3930818313fb\") " pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" Jul 03 14:43:22.986979 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:43:22.986883 2576 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-4cf4ef-serving-cert: secret "maas-api-e2e-api-a-4cf4ef-serving-cert" not found Jul 03 14:43:22.986979 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:43:22.986959 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb4a096-04ee-4d64-a99b-3930818313fb-maas-api-tls podName:bbb4a096-04ee-4d64-a99b-3930818313fb nodeName:}" failed. No retries permitted until 2026-07-03 14:43:23.486941566 +0000 UTC m=+2081.843226164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bbb4a096-04ee-4d64-a99b-3930818313fb-maas-api-tls") pod "maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" (UID: "bbb4a096-04ee-4d64-a99b-3930818313fb") : secret "maas-api-e2e-api-a-4cf4ef-serving-cert" not found Jul 03 14:43:22.987378 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:22.987360 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb4a096-04ee-4d64-a99b-3930818313fb-openshift-service-ca\") pod \"maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv\" (UID: \"bbb4a096-04ee-4d64-a99b-3930818313fb\") " pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" Jul 03 14:43:22.997695 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:22.997650 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxmpv\" (UniqueName: \"kubernetes.io/projected/bbb4a096-04ee-4d64-a99b-3930818313fb-kube-api-access-zxmpv\") pod \"maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv\" (UID: \"bbb4a096-04ee-4d64-a99b-3930818313fb\") " pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" Jul 03 14:43:23.492478 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:23.492442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbb4a096-04ee-4d64-a99b-3930818313fb-maas-api-tls\") pod \"maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv\" (UID: \"bbb4a096-04ee-4d64-a99b-3930818313fb\") " pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" Jul 03 14:43:23.495065 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:23.495035 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbb4a096-04ee-4d64-a99b-3930818313fb-maas-api-tls\") pod \"maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv\" (UID: \"bbb4a096-04ee-4d64-a99b-3930818313fb\") " pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" Jul 03 14:43:23.760279 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:23.760193 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" Jul 03 14:43:23.906290 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:23.906263 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv"] Jul 03 14:43:23.908079 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:43:23.908050 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb4a096_04ee_4d64_a99b_3930818313fb.slice/crio-5d767433bffd3c26c33ffe79d16ead77022158bd567ae5683faf194656e37f1d WatchSource:0}: Error finding container 5d767433bffd3c26c33ffe79d16ead77022158bd567ae5683faf194656e37f1d: Status 404 returned error can't find the container with id 5d767433bffd3c26c33ffe79d16ead77022158bd567ae5683faf194656e37f1d Jul 03 14:43:24.767415 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:24.767331 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" event={"ID":"bbb4a096-04ee-4d64-a99b-3930818313fb","Type":"ContainerStarted","Data":"8f65ef2650fcd04878e30a8dbc7478c43c6c7a4af2cedb3a6c0ab2841889c35a"} Jul 03 14:43:24.767415 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:24.767381 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" event={"ID":"bbb4a096-04ee-4d64-a99b-3930818313fb","Type":"ContainerStarted","Data":"5d767433bffd3c26c33ffe79d16ead77022158bd567ae5683faf194656e37f1d"} Jul 03 14:43:24.767611 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:24.767458 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" Jul 03 14:43:24.802829 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:24.802773 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" podStartSLOduration=2.422610622 podStartE2EDuration="2.802755582s" totalImagesPullingTime="380.14496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:43:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:43:24.80087076 +0000 UTC m=+2083.157155389" watchObservedRunningTime="2026-07-03 14:43:24.802755582 +0000 UTC m=+2083.159040202" Jul 03 14:43:30.779232 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:30.779200 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" Jul 03 14:43:31.510398 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:31.509891 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5"] Jul 03 14:43:31.513976 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:31.513953 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" Jul 03 14:43:31.516618 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:31.516591 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-86730a-serving-cert\"" Jul 03 14:43:31.526842 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:31.526771 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5"] Jul 03 14:43:31.570713 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:31.570647 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-maas-api-tls\") pod \"maas-api-e2e-api-b-86730a-fb4474887-rqxr5\" (UID: \"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1\") " pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" Jul 03 14:43:31.570903 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:31.570724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-openshift-service-ca\") pod \"maas-api-e2e-api-b-86730a-fb4474887-rqxr5\" (UID: \"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1\") " pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" Jul 03 14:43:31.570903 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:31.570771 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb6ss\" (UniqueName: \"kubernetes.io/projected/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-kube-api-access-cb6ss\") pod \"maas-api-e2e-api-b-86730a-fb4474887-rqxr5\" (UID: \"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1\") " pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" Jul 03 14:43:31.672062 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:31.672023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-openshift-service-ca\") pod \"maas-api-e2e-api-b-86730a-fb4474887-rqxr5\" (UID: \"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1\") " pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" Jul 03 14:43:31.672251 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:31.672087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb6ss\" (UniqueName: \"kubernetes.io/projected/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-kube-api-access-cb6ss\") pod \"maas-api-e2e-api-b-86730a-fb4474887-rqxr5\" (UID: \"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1\") " pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" Jul 03 14:43:31.672251 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:31.672158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-maas-api-tls\") pod \"maas-api-e2e-api-b-86730a-fb4474887-rqxr5\" (UID: \"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1\") " pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" Jul 03 14:43:31.672357 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:43:31.672257 2576 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-86730a-serving-cert: secret "maas-api-e2e-api-b-86730a-serving-cert" not found Jul 03 14:43:31.672357 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:43:31.672313 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-maas-api-tls podName:25c4e7fc-5cae-42d0-ac64-804dd6e4eae1 nodeName:}" failed. No retries permitted until 2026-07-03 14:43:32.172297988 +0000 UTC m=+2090.528582585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-maas-api-tls") pod "maas-api-e2e-api-b-86730a-fb4474887-rqxr5" (UID: "25c4e7fc-5cae-42d0-ac64-804dd6e4eae1") : secret "maas-api-e2e-api-b-86730a-serving-cert" not found Jul 03 14:43:31.672806 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:31.672785 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-openshift-service-ca\") pod \"maas-api-e2e-api-b-86730a-fb4474887-rqxr5\" (UID: \"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1\") " pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" Jul 03 14:43:31.681615 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:31.681584 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb6ss\" (UniqueName: \"kubernetes.io/projected/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-kube-api-access-cb6ss\") pod \"maas-api-e2e-api-b-86730a-fb4474887-rqxr5\" (UID: \"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1\") " pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" Jul 03 14:43:32.176722 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:32.176690 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-maas-api-tls\") pod \"maas-api-e2e-api-b-86730a-fb4474887-rqxr5\" (UID: \"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1\") " pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" Jul 03 14:43:32.179399 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:32.179372 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-maas-api-tls\") pod \"maas-api-e2e-api-b-86730a-fb4474887-rqxr5\" (UID: \"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1\") " pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" Jul 03 14:43:32.428722 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:32.428602 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" Jul 03 14:43:32.648520 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:32.648477 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5"] Jul 03 14:43:32.650315 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:43:32.650281 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c4e7fc_5cae_42d0_ac64_804dd6e4eae1.slice/crio-612ee62d38a413b537be16dcda92205eda6d8b02db2d464a1238a0a0309d1de6 WatchSource:0}: Error finding container 612ee62d38a413b537be16dcda92205eda6d8b02db2d464a1238a0a0309d1de6: Status 404 returned error can't find the container with id 612ee62d38a413b537be16dcda92205eda6d8b02db2d464a1238a0a0309d1de6 Jul 03 14:43:32.806462 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:32.806428 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" event={"ID":"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1","Type":"ContainerStarted","Data":"612ee62d38a413b537be16dcda92205eda6d8b02db2d464a1238a0a0309d1de6"} Jul 03 14:43:33.812981 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:33.812934 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" event={"ID":"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1","Type":"ContainerStarted","Data":"6118b0802f07eb8afb513f399482dcfa11e4e26addc8c6262f295c3d9db8f49f"} Jul 03 14:43:33.813407 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:33.813036 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" Jul 03 14:43:33.832720 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:33.832624 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" podStartSLOduration=2.4741263780000002 podStartE2EDuration="2.832602183s" totalImagesPullingTime="358.475805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:43:33.830596017 +0000 UTC m=+2092.186880639" watchObservedRunningTime="2026-07-03 14:43:33.832602183 +0000 UTC m=+2092.188886804" Jul 03 14:43:37.792863 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:37.792822 2576 patch_prober.go:28] interesting pod/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 03 14:43:37.793264 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:37.792890 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" podUID="ace7d172-70ea-470c-b659-73318430b0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 03 14:43:38.839242 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:38.839157 2576 generic.go:356] "Generic (PLEG): container finished" podID="ace7d172-70ea-470c-b659-73318430b0bd" containerID="8decb76e0ea6c76cb3ef93ea92edca012e58d47596692ffa99e0046d124b5a65" exitCode=0 Jul 03 14:43:38.839242 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:38.839218 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" event={"ID":"ace7d172-70ea-470c-b659-73318430b0bd","Type":"ContainerDied","Data":"8decb76e0ea6c76cb3ef93ea92edca012e58d47596692ffa99e0046d124b5a65"} Jul 03 14:43:38.839706 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:38.839618 2576 scope.go:117] "RemoveContainer" containerID="8decb76e0ea6c76cb3ef93ea92edca012e58d47596692ffa99e0046d124b5a65" Jul 03 14:43:38.840586 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:38.840557 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:43:38.840671 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:38.840642 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:43:38.840718 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:38.840702 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 03 14:43:39.830980 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:39.830937 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" Jul 03 14:43:39.853458 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:39.853423 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" event={"ID":"ace7d172-70ea-470c-b659-73318430b0bd","Type":"ContainerStarted","Data":"90632a3c2771ee47990d488c0658a60913b69660dbab45306c7be464ecd1ec6b"} Jul 03 14:43:40.476372 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:40.476326 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:40.476557 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:40.476539 2576 patch_prober.go:28] interesting pod/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 03 14:43:40.476602 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:40.476586 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" podUID="ace7d172-70ea-470c-b659-73318430b0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 03 14:43:41.476995 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:41.476959 2576 patch_prober.go:28] interesting pod/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 03 14:43:41.477402 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:41.477018 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" podUID="ace7d172-70ea-470c-b659-73318430b0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 03 14:43:42.235685 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:42.219750 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:43:42.247955 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:42.247924 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:43:42.248756 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:42.248727 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:43:42.252243 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:42.252219 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:43:42.476929 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:42.476888 2576 patch_prober.go:28] interesting pod/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 03 14:43:42.477109 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:42.476960 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" podUID="ace7d172-70ea-470c-b659-73318430b0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 03 14:43:43.476475 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:43.476434 2576 patch_prober.go:28] interesting pod/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 03 14:43:43.476747 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:43.476501 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" podUID="ace7d172-70ea-470c-b659-73318430b0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 03 14:43:44.480480 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:44.480453 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:44.480923 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:44.480726 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:44.481403 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:44.481387 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:44.498582 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:44.498550 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv"] Jul 03 14:43:44.498830 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:44.498799 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" podUID="bbb4a096-04ee-4d64-a99b-3930818313fb" containerName="maas-api" containerID="cri-o://8f65ef2650fcd04878e30a8dbc7478c43c6c7a4af2cedb3a6c0ab2841889c35a" gracePeriod=30 Jul 03 14:43:44.746410 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:44.746387 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" Jul 03 14:43:44.878927 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:44.878889 2576 generic.go:356] "Generic (PLEG): container finished" podID="bbb4a096-04ee-4d64-a99b-3930818313fb" containerID="8f65ef2650fcd04878e30a8dbc7478c43c6c7a4af2cedb3a6c0ab2841889c35a" exitCode=0 Jul 03 14:43:44.878927 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:44.878932 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" Jul 03 14:43:44.879157 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:44.878973 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" event={"ID":"bbb4a096-04ee-4d64-a99b-3930818313fb","Type":"ContainerDied","Data":"8f65ef2650fcd04878e30a8dbc7478c43c6c7a4af2cedb3a6c0ab2841889c35a"} Jul 03 14:43:44.879157 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:44.879021 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv" event={"ID":"bbb4a096-04ee-4d64-a99b-3930818313fb","Type":"ContainerDied","Data":"5d767433bffd3c26c33ffe79d16ead77022158bd567ae5683faf194656e37f1d"} Jul 03 14:43:44.879157 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:44.879043 2576 scope.go:117] "RemoveContainer" containerID="8f65ef2650fcd04878e30a8dbc7478c43c6c7a4af2cedb3a6c0ab2841889c35a" Jul 03 14:43:44.888003 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:44.887986 2576 scope.go:117] "RemoveContainer" containerID="8f65ef2650fcd04878e30a8dbc7478c43c6c7a4af2cedb3a6c0ab2841889c35a" Jul 03 14:43:44.888251 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:43:44.888233 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f65ef2650fcd04878e30a8dbc7478c43c6c7a4af2cedb3a6c0ab2841889c35a\": container with ID starting with 8f65ef2650fcd04878e30a8dbc7478c43c6c7a4af2cedb3a6c0ab2841889c35a not found: ID does not exist" containerID="8f65ef2650fcd04878e30a8dbc7478c43c6c7a4af2cedb3a6c0ab2841889c35a" Jul 03 14:43:44.888298 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:44.888260 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f65ef2650fcd04878e30a8dbc7478c43c6c7a4af2cedb3a6c0ab2841889c35a"} err="failed to get container status \"8f65ef2650fcd04878e30a8dbc7478c43c6c7a4af2cedb3a6c0ab2841889c35a\": rpc error: code = NotFound desc = could not find container \"8f65ef2650fcd04878e30a8dbc7478c43c6c7a4af2cedb3a6c0ab2841889c35a\": container with ID starting with 8f65ef2650fcd04878e30a8dbc7478c43c6c7a4af2cedb3a6c0ab2841889c35a not found: ID does not exist" Jul 03 14:43:44.900584 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:44.900561 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxmpv\" (UniqueName: \"kubernetes.io/projected/bbb4a096-04ee-4d64-a99b-3930818313fb-kube-api-access-zxmpv\") pod \"bbb4a096-04ee-4d64-a99b-3930818313fb\" (UID: \"bbb4a096-04ee-4d64-a99b-3930818313fb\") " Jul 03 14:43:44.900678 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:44.900601 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbb4a096-04ee-4d64-a99b-3930818313fb-maas-api-tls\") pod \"bbb4a096-04ee-4d64-a99b-3930818313fb\" (UID: \"bbb4a096-04ee-4d64-a99b-3930818313fb\") " Jul 03 14:43:44.900678 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:44.900638 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb4a096-04ee-4d64-a99b-3930818313fb-openshift-service-ca\") pod \"bbb4a096-04ee-4d64-a99b-3930818313fb\" (UID: \"bbb4a096-04ee-4d64-a99b-3930818313fb\") " Jul 03 14:43:44.901021 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:44.901000 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbb4a096-04ee-4d64-a99b-3930818313fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bbb4a096-04ee-4d64-a99b-3930818313fb" (UID: "bbb4a096-04ee-4d64-a99b-3930818313fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:43:44.902829 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:44.902810 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbb4a096-04ee-4d64-a99b-3930818313fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bbb4a096-04ee-4d64-a99b-3930818313fb" (UID: "bbb4a096-04ee-4d64-a99b-3930818313fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 14:43:44.902926 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:44.902828 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb4a096-04ee-4d64-a99b-3930818313fb-kube-api-access-zxmpv" (OuterVolumeSpecName: "kube-api-access-zxmpv") pod "bbb4a096-04ee-4d64-a99b-3930818313fb" (UID: "bbb4a096-04ee-4d64-a99b-3930818313fb"). InnerVolumeSpecName "kube-api-access-zxmpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:43:45.002286 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:45.002196 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxmpv\" (UniqueName: \"kubernetes.io/projected/bbb4a096-04ee-4d64-a99b-3930818313fb-kube-api-access-zxmpv\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:45.002286 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:45.002225 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbb4a096-04ee-4d64-a99b-3930818313fb-maas-api-tls\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:45.002286 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:45.002234 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb4a096-04ee-4d64-a99b-3930818313fb-openshift-service-ca\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:45.202104 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:45.202069 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv"] Jul 03 14:43:45.205497 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:45.205471 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-4cf4ef-85c8fcb9f9-bffdv"] Jul 03 14:43:46.167222 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:46.167192 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbb4a096-04ee-4d64-a99b-3930818313fb" path="/var/lib/kubelet/pods/bbb4a096-04ee-4d64-a99b-3930818313fb/volumes" Jul 03 14:43:50.547993 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:50.547960 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w"] Jul 03 14:43:50.548472 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:50.548177 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" podUID="ace7d172-70ea-470c-b659-73318430b0bd" containerName="istio-proxy" containerID="cri-o://90632a3c2771ee47990d488c0658a60913b69660dbab45306c7be464ecd1ec6b" gracePeriod=30 Jul 03 14:43:51.589088 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:51.589051 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5"] Jul 03 14:43:51.589486 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:51.589293 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" podUID="25c4e7fc-5cae-42d0-ac64-804dd6e4eae1" containerName="maas-api" containerID="cri-o://6118b0802f07eb8afb513f399482dcfa11e4e26addc8c6262f295c3d9db8f49f" gracePeriod=30 Jul 03 14:43:51.842046 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:51.841991 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" Jul 03 14:43:51.912699 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:51.912645 2576 generic.go:356] "Generic (PLEG): container finished" podID="25c4e7fc-5cae-42d0-ac64-804dd6e4eae1" containerID="6118b0802f07eb8afb513f399482dcfa11e4e26addc8c6262f295c3d9db8f49f" exitCode=0 Jul 03 14:43:51.912875 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:51.912706 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" Jul 03 14:43:51.912875 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:51.912698 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" event={"ID":"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1","Type":"ContainerDied","Data":"6118b0802f07eb8afb513f399482dcfa11e4e26addc8c6262f295c3d9db8f49f"} Jul 03 14:43:51.912875 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:51.912830 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5" event={"ID":"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1","Type":"ContainerDied","Data":"612ee62d38a413b537be16dcda92205eda6d8b02db2d464a1238a0a0309d1de6"} Jul 03 14:43:51.912875 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:51.912856 2576 scope.go:117] "RemoveContainer" containerID="6118b0802f07eb8afb513f399482dcfa11e4e26addc8c6262f295c3d9db8f49f" Jul 03 14:43:51.922071 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:51.922047 2576 scope.go:117] "RemoveContainer" containerID="6118b0802f07eb8afb513f399482dcfa11e4e26addc8c6262f295c3d9db8f49f" Jul 03 14:43:51.922423 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:43:51.922400 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6118b0802f07eb8afb513f399482dcfa11e4e26addc8c6262f295c3d9db8f49f\": container with ID starting with 6118b0802f07eb8afb513f399482dcfa11e4e26addc8c6262f295c3d9db8f49f not found: ID does not exist" containerID="6118b0802f07eb8afb513f399482dcfa11e4e26addc8c6262f295c3d9db8f49f" Jul 03 14:43:51.922566 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:51.922543 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6118b0802f07eb8afb513f399482dcfa11e4e26addc8c6262f295c3d9db8f49f"} err="failed to get container status \"6118b0802f07eb8afb513f399482dcfa11e4e26addc8c6262f295c3d9db8f49f\": rpc error: code = NotFound desc = could not find container \"6118b0802f07eb8afb513f399482dcfa11e4e26addc8c6262f295c3d9db8f49f\": container with ID starting with 6118b0802f07eb8afb513f399482dcfa11e4e26addc8c6262f295c3d9db8f49f not found: ID does not exist" Jul 03 14:43:51.969097 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:51.969066 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-maas-api-tls\") pod \"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1\" (UID: \"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1\") " Jul 03 14:43:51.969266 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:51.969170 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb6ss\" (UniqueName: \"kubernetes.io/projected/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-kube-api-access-cb6ss\") pod \"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1\" (UID: \"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1\") " Jul 03 14:43:51.969266 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:51.969193 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-openshift-service-ca\") pod \"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1\" (UID: \"25c4e7fc-5cae-42d0-ac64-804dd6e4eae1\") " Jul 03 14:43:51.969603 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:51.969576 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "25c4e7fc-5cae-42d0-ac64-804dd6e4eae1" (UID: "25c4e7fc-5cae-42d0-ac64-804dd6e4eae1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:43:51.971339 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:51.971317 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "25c4e7fc-5cae-42d0-ac64-804dd6e4eae1" (UID: "25c4e7fc-5cae-42d0-ac64-804dd6e4eae1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 14:43:51.971470 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:51.971368 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-kube-api-access-cb6ss" (OuterVolumeSpecName: "kube-api-access-cb6ss") pod "25c4e7fc-5cae-42d0-ac64-804dd6e4eae1" (UID: "25c4e7fc-5cae-42d0-ac64-804dd6e4eae1"). InnerVolumeSpecName "kube-api-access-cb6ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:43:52.070158 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:52.070122 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb6ss\" (UniqueName: \"kubernetes.io/projected/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-kube-api-access-cb6ss\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:52.070158 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:52.070152 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-openshift-service-ca\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:52.070158 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:52.070162 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1-maas-api-tls\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:52.229219 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:52.229180 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5"] Jul 03 14:43:52.232732 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:52.232706 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-86730a-fb4474887-rqxr5"] Jul 03 14:43:54.168511 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:54.168478 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25c4e7fc-5cae-42d0-ac64-804dd6e4eae1" path="/var/lib/kubelet/pods/25c4e7fc-5cae-42d0-ac64-804dd6e4eae1/volumes" Jul 03 14:43:55.805719 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.805690 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:55.903557 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.903476 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-workload-certs\") pod \"ace7d172-70ea-470c-b659-73318430b0bd\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " Jul 03 14:43:55.903557 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.903530 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace7d172-70ea-470c-b659-73318430b0bd-istio-token\") pod \"ace7d172-70ea-470c-b659-73318430b0bd\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " Jul 03 14:43:55.903557 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.903552 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace7d172-70ea-470c-b659-73318430b0bd-istio-podinfo\") pod \"ace7d172-70ea-470c-b659-73318430b0bd\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " Jul 03 14:43:55.903848 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.903594 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-credential-socket\") pod \"ace7d172-70ea-470c-b659-73318430b0bd\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " Jul 03 14:43:55.903848 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.903617 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-istio-envoy\") pod \"ace7d172-70ea-470c-b659-73318430b0bd\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " Jul 03 14:43:55.903848 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.903640 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6g58v\" (UniqueName: \"kubernetes.io/projected/ace7d172-70ea-470c-b659-73318430b0bd-kube-api-access-6g58v\") pod \"ace7d172-70ea-470c-b659-73318430b0bd\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " Jul 03 14:43:55.903848 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.903695 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-istio-data\") pod \"ace7d172-70ea-470c-b659-73318430b0bd\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " Jul 03 14:43:55.903848 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.903723 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-workload-socket\") pod \"ace7d172-70ea-470c-b659-73318430b0bd\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " Jul 03 14:43:55.903848 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.903787 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace7d172-70ea-470c-b659-73318430b0bd-istiod-ca-cert\") pod \"ace7d172-70ea-470c-b659-73318430b0bd\" (UID: \"ace7d172-70ea-470c-b659-73318430b0bd\") " Jul 03 14:43:55.903848 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.903801 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ace7d172-70ea-470c-b659-73318430b0bd" (UID: "ace7d172-70ea-470c-b659-73318430b0bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:43:55.904199 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.904009 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ace7d172-70ea-470c-b659-73318430b0bd" (UID: "ace7d172-70ea-470c-b659-73318430b0bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:43:55.904199 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.904088 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-credential-socket\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:55.904199 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.904116 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-workload-certs\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:55.904582 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.904275 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace7d172-70ea-470c-b659-73318430b0bd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ace7d172-70ea-470c-b659-73318430b0bd" (UID: "ace7d172-70ea-470c-b659-73318430b0bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:43:55.904582 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.904573 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ace7d172-70ea-470c-b659-73318430b0bd" (UID: "ace7d172-70ea-470c-b659-73318430b0bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:43:55.904812 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.904792 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-istio-data" (OuterVolumeSpecName: "istio-data") pod "ace7d172-70ea-470c-b659-73318430b0bd" (UID: "ace7d172-70ea-470c-b659-73318430b0bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:43:55.905966 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.905943 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ace7d172-70ea-470c-b659-73318430b0bd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ace7d172-70ea-470c-b659-73318430b0bd" (UID: "ace7d172-70ea-470c-b659-73318430b0bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 03 14:43:55.905966 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.905946 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ace7d172-70ea-470c-b659-73318430b0bd" (UID: "ace7d172-70ea-470c-b659-73318430b0bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 14:43:55.906466 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.906439 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace7d172-70ea-470c-b659-73318430b0bd-istio-token" (OuterVolumeSpecName: "istio-token") pod "ace7d172-70ea-470c-b659-73318430b0bd" (UID: "ace7d172-70ea-470c-b659-73318430b0bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:43:55.906565 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.906439 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace7d172-70ea-470c-b659-73318430b0bd-kube-api-access-6g58v" (OuterVolumeSpecName: "kube-api-access-6g58v") pod "ace7d172-70ea-470c-b659-73318430b0bd" (UID: "ace7d172-70ea-470c-b659-73318430b0bd"). InnerVolumeSpecName "kube-api-access-6g58v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:43:55.939331 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.939298 2576 generic.go:356] "Generic (PLEG): container finished" podID="ace7d172-70ea-470c-b659-73318430b0bd" containerID="90632a3c2771ee47990d488c0658a60913b69660dbab45306c7be464ecd1ec6b" exitCode=0 Jul 03 14:43:55.939474 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.939354 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" Jul 03 14:43:55.939474 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.939387 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" event={"ID":"ace7d172-70ea-470c-b659-73318430b0bd","Type":"ContainerDied","Data":"90632a3c2771ee47990d488c0658a60913b69660dbab45306c7be464ecd1ec6b"} Jul 03 14:43:55.939474 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.939439 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w" event={"ID":"ace7d172-70ea-470c-b659-73318430b0bd","Type":"ContainerDied","Data":"32b99e7859b02bf24ee6dbc0d62d13224e89b3c1b5e5c791bdb59bba6a28b375"} Jul 03 14:43:55.939474 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.939471 2576 scope.go:117] "RemoveContainer" containerID="90632a3c2771ee47990d488c0658a60913b69660dbab45306c7be464ecd1ec6b" Jul 03 14:43:55.948821 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.948804 2576 scope.go:117] "RemoveContainer" containerID="8decb76e0ea6c76cb3ef93ea92edca012e58d47596692ffa99e0046d124b5a65" Jul 03 14:43:55.956719 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.956700 2576 scope.go:117] "RemoveContainer" containerID="90632a3c2771ee47990d488c0658a60913b69660dbab45306c7be464ecd1ec6b" Jul 03 14:43:55.956961 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:43:55.956944 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90632a3c2771ee47990d488c0658a60913b69660dbab45306c7be464ecd1ec6b\": container with ID starting with 90632a3c2771ee47990d488c0658a60913b69660dbab45306c7be464ecd1ec6b not found: ID does not exist" containerID="90632a3c2771ee47990d488c0658a60913b69660dbab45306c7be464ecd1ec6b" Jul 03 14:43:55.957010 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.956969 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90632a3c2771ee47990d488c0658a60913b69660dbab45306c7be464ecd1ec6b"} err="failed to get container status \"90632a3c2771ee47990d488c0658a60913b69660dbab45306c7be464ecd1ec6b\": rpc error: code = NotFound desc = could not find container \"90632a3c2771ee47990d488c0658a60913b69660dbab45306c7be464ecd1ec6b\": container with ID starting with 90632a3c2771ee47990d488c0658a60913b69660dbab45306c7be464ecd1ec6b not found: ID does not exist" Jul 03 14:43:55.957010 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.956988 2576 scope.go:117] "RemoveContainer" containerID="8decb76e0ea6c76cb3ef93ea92edca012e58d47596692ffa99e0046d124b5a65" Jul 03 14:43:55.957236 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:43:55.957221 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8decb76e0ea6c76cb3ef93ea92edca012e58d47596692ffa99e0046d124b5a65\": container with ID starting with 8decb76e0ea6c76cb3ef93ea92edca012e58d47596692ffa99e0046d124b5a65 not found: ID does not exist" containerID="8decb76e0ea6c76cb3ef93ea92edca012e58d47596692ffa99e0046d124b5a65" Jul 03 14:43:55.957297 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.957253 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8decb76e0ea6c76cb3ef93ea92edca012e58d47596692ffa99e0046d124b5a65"} err="failed to get container status \"8decb76e0ea6c76cb3ef93ea92edca012e58d47596692ffa99e0046d124b5a65\": rpc error: code = NotFound desc = could not find container \"8decb76e0ea6c76cb3ef93ea92edca012e58d47596692ffa99e0046d124b5a65\": container with ID starting with 8decb76e0ea6c76cb3ef93ea92edca012e58d47596692ffa99e0046d124b5a65 not found: ID does not exist" Jul 03 14:43:55.962012 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.961988 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w"] Jul 03 14:43:55.965379 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:55.965360 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4cf4ef-openshift-default-568d7fbc-9744w"] Jul 03 14:43:56.005467 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:56.005427 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace7d172-70ea-470c-b659-73318430b0bd-istiod-ca-cert\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:56.005467 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:56.005454 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace7d172-70ea-470c-b659-73318430b0bd-istio-token\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:56.005467 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:56.005464 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace7d172-70ea-470c-b659-73318430b0bd-istio-podinfo\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:56.005467 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:56.005473 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-istio-envoy\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:56.005741 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:56.005481 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g58v\" (UniqueName: \"kubernetes.io/projected/ace7d172-70ea-470c-b659-73318430b0bd-kube-api-access-6g58v\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:56.005741 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:56.005492 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-istio-data\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:56.005741 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:56.005502 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace7d172-70ea-470c-b659-73318430b0bd-workload-socket\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:43:56.167617 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:43:56.167531 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ace7d172-70ea-470c-b659-73318430b0bd" path="/var/lib/kubelet/pods/ace7d172-70ea-470c-b659-73318430b0bd/volumes" Jul 03 14:44:01.681305 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.681266 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x"] Jul 03 14:44:01.681706 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.681609 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbb4a096-04ee-4d64-a99b-3930818313fb" containerName="maas-api" Jul 03 14:44:01.681706 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.681620 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbb4a096-04ee-4d64-a99b-3930818313fb" containerName="maas-api" Jul 03 14:44:01.681706 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.681629 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25c4e7fc-5cae-42d0-ac64-804dd6e4eae1" containerName="maas-api" Jul 03 14:44:01.681706 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.681635 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="25c4e7fc-5cae-42d0-ac64-804dd6e4eae1" containerName="maas-api" Jul 03 14:44:01.681706 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.681644 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ace7d172-70ea-470c-b659-73318430b0bd" containerName="istio-proxy" Jul 03 14:44:01.681706 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.681669 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ace7d172-70ea-470c-b659-73318430b0bd" containerName="istio-proxy" Jul 03 14:44:01.681706 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.681680 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ace7d172-70ea-470c-b659-73318430b0bd" containerName="istio-proxy" Jul 03 14:44:01.681706 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.681688 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ace7d172-70ea-470c-b659-73318430b0bd" containerName="istio-proxy" Jul 03 14:44:01.681978 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.681738 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ace7d172-70ea-470c-b659-73318430b0bd" containerName="istio-proxy" Jul 03 14:44:01.681978 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.681745 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="25c4e7fc-5cae-42d0-ac64-804dd6e4eae1" containerName="maas-api" Jul 03 14:44:01.681978 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.681752 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbb4a096-04ee-4d64-a99b-3930818313fb" containerName="maas-api" Jul 03 14:44:01.686163 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.686143 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" Jul 03 14:44:01.688889 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.688869 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-5f1e04-serving-cert\"" Jul 03 14:44:01.695354 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.695325 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x"] Jul 03 14:44:01.761827 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.761796 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q54z\" (UniqueName: \"kubernetes.io/projected/19c702a4-76c3-49a8-b712-205767fb717c-kube-api-access-8q54z\") pod \"maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x\" (UID: \"19c702a4-76c3-49a8-b712-205767fb717c\") " pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" Jul 03 14:44:01.762006 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.761869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c702a4-76c3-49a8-b712-205767fb717c-maas-api-tls\") pod \"maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x\" (UID: \"19c702a4-76c3-49a8-b712-205767fb717c\") " pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" Jul 03 14:44:01.762006 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.761900 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c702a4-76c3-49a8-b712-205767fb717c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x\" (UID: \"19c702a4-76c3-49a8-b712-205767fb717c\") " pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" Jul 03 14:44:01.863243 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.863203 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c702a4-76c3-49a8-b712-205767fb717c-maas-api-tls\") pod \"maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x\" (UID: \"19c702a4-76c3-49a8-b712-205767fb717c\") " pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" Jul 03 14:44:01.863243 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.863245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c702a4-76c3-49a8-b712-205767fb717c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x\" (UID: \"19c702a4-76c3-49a8-b712-205767fb717c\") " pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" Jul 03 14:44:01.863476 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.863327 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q54z\" (UniqueName: \"kubernetes.io/projected/19c702a4-76c3-49a8-b712-205767fb717c-kube-api-access-8q54z\") pod \"maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x\" (UID: \"19c702a4-76c3-49a8-b712-205767fb717c\") " pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" Jul 03 14:44:01.863476 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:44:01.863370 2576 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-5f1e04-serving-cert: secret "maas-api-e2e-shared-a-5f1e04-serving-cert" not found Jul 03 14:44:01.863476 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:44:01.863439 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19c702a4-76c3-49a8-b712-205767fb717c-maas-api-tls podName:19c702a4-76c3-49a8-b712-205767fb717c nodeName:}" failed. No retries permitted until 2026-07-03 14:44:02.363419034 +0000 UTC m=+2120.719703645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/19c702a4-76c3-49a8-b712-205767fb717c-maas-api-tls") pod "maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" (UID: "19c702a4-76c3-49a8-b712-205767fb717c") : secret "maas-api-e2e-shared-a-5f1e04-serving-cert" not found Jul 03 14:44:01.864132 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.864107 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c702a4-76c3-49a8-b712-205767fb717c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x\" (UID: \"19c702a4-76c3-49a8-b712-205767fb717c\") " pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" Jul 03 14:44:01.874461 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:01.874439 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q54z\" (UniqueName: \"kubernetes.io/projected/19c702a4-76c3-49a8-b712-205767fb717c-kube-api-access-8q54z\") pod \"maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x\" (UID: \"19c702a4-76c3-49a8-b712-205767fb717c\") " pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" Jul 03 14:44:02.368164 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:02.368120 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c702a4-76c3-49a8-b712-205767fb717c-maas-api-tls\") pod \"maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x\" (UID: \"19c702a4-76c3-49a8-b712-205767fb717c\") " pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" Jul 03 14:44:02.370724 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:02.370699 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c702a4-76c3-49a8-b712-205767fb717c-maas-api-tls\") pod \"maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x\" (UID: \"19c702a4-76c3-49a8-b712-205767fb717c\") " pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" Jul 03 14:44:02.599159 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:02.599121 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" Jul 03 14:44:02.751546 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:02.751519 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x"] Jul 03 14:44:02.754197 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:44:02.754158 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c702a4_76c3_49a8_b712_205767fb717c.slice/crio-8c56d4c97771ce13c018decae65329575fd21c09718f717cd5b527ecfdaf7064 WatchSource:0}: Error finding container 8c56d4c97771ce13c018decae65329575fd21c09718f717cd5b527ecfdaf7064: Status 404 returned error can't find the container with id 8c56d4c97771ce13c018decae65329575fd21c09718f717cd5b527ecfdaf7064 Jul 03 14:44:02.974104 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:02.974020 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" event={"ID":"19c702a4-76c3-49a8-b712-205767fb717c","Type":"ContainerStarted","Data":"8c56d4c97771ce13c018decae65329575fd21c09718f717cd5b527ecfdaf7064"} Jul 03 14:44:03.980623 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:03.980586 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" event={"ID":"19c702a4-76c3-49a8-b712-205767fb717c","Type":"ContainerStarted","Data":"2e4941d098420cc3f6c618948f3919ff6aef471b2ec5aebcbf3d753fc3bce6c9"} Jul 03 14:44:03.981009 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:03.980824 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" Jul 03 14:44:04.011386 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:04.011333 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" podStartSLOduration=2.625608246 podStartE2EDuration="3.011315282s" totalImagesPullingTime="385.707036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:44:04.00840921 +0000 UTC m=+2122.364693830" watchObservedRunningTime="2026-07-03 14:44:04.011315282 +0000 UTC m=+2122.367599902" Jul 03 14:44:09.714595 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:09.714562 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg"] Jul 03 14:44:09.715183 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:09.715153 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ace7d172-70ea-470c-b659-73318430b0bd" containerName="istio-proxy" Jul 03 14:44:09.718773 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:09.718749 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" Jul 03 14:44:09.721719 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:09.721697 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-28686a-serving-cert\"" Jul 03 14:44:09.731434 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:09.731387 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg"] Jul 03 14:44:09.832289 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:09.832249 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af94e282-da48-4322-9d14-f7206bd8a824-maas-api-tls\") pod \"maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg\" (UID: \"af94e282-da48-4322-9d14-f7206bd8a824\") " pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" Jul 03 14:44:09.832472 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:09.832320 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af94e282-da48-4322-9d14-f7206bd8a824-openshift-service-ca\") pod \"maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg\" (UID: \"af94e282-da48-4322-9d14-f7206bd8a824\") " pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" Jul 03 14:44:09.832472 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:09.832349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkq65\" (UniqueName: \"kubernetes.io/projected/af94e282-da48-4322-9d14-f7206bd8a824-kube-api-access-nkq65\") pod \"maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg\" (UID: \"af94e282-da48-4322-9d14-f7206bd8a824\") " pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" Jul 03 14:44:09.933098 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:09.933061 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af94e282-da48-4322-9d14-f7206bd8a824-maas-api-tls\") pod \"maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg\" (UID: \"af94e282-da48-4322-9d14-f7206bd8a824\") " pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" Jul 03 14:44:09.933301 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:09.933120 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af94e282-da48-4322-9d14-f7206bd8a824-openshift-service-ca\") pod \"maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg\" (UID: \"af94e282-da48-4322-9d14-f7206bd8a824\") " pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" Jul 03 14:44:09.933301 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:09.933149 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkq65\" (UniqueName: \"kubernetes.io/projected/af94e282-da48-4322-9d14-f7206bd8a824-kube-api-access-nkq65\") pod \"maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg\" (UID: \"af94e282-da48-4322-9d14-f7206bd8a824\") " pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" Jul 03 14:44:09.933301 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:44:09.933231 2576 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-28686a-serving-cert: secret "maas-api-e2e-shared-b-28686a-serving-cert" not found Jul 03 14:44:09.933469 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:44:09.933314 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af94e282-da48-4322-9d14-f7206bd8a824-maas-api-tls podName:af94e282-da48-4322-9d14-f7206bd8a824 nodeName:}" failed. No retries permitted until 2026-07-03 14:44:10.433291851 +0000 UTC m=+2128.789576456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af94e282-da48-4322-9d14-f7206bd8a824-maas-api-tls") pod "maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" (UID: "af94e282-da48-4322-9d14-f7206bd8a824") : secret "maas-api-e2e-shared-b-28686a-serving-cert" not found Jul 03 14:44:09.933936 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:09.933914 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af94e282-da48-4322-9d14-f7206bd8a824-openshift-service-ca\") pod \"maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg\" (UID: \"af94e282-da48-4322-9d14-f7206bd8a824\") " pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" Jul 03 14:44:09.944783 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:09.944756 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkq65\" (UniqueName: \"kubernetes.io/projected/af94e282-da48-4322-9d14-f7206bd8a824-kube-api-access-nkq65\") pod \"maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg\" (UID: \"af94e282-da48-4322-9d14-f7206bd8a824\") " pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" Jul 03 14:44:09.991896 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:09.991820 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" Jul 03 14:44:10.437555 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:10.437513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af94e282-da48-4322-9d14-f7206bd8a824-maas-api-tls\") pod \"maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg\" (UID: \"af94e282-da48-4322-9d14-f7206bd8a824\") " pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" Jul 03 14:44:10.440175 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:10.440148 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af94e282-da48-4322-9d14-f7206bd8a824-maas-api-tls\") pod \"maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg\" (UID: \"af94e282-da48-4322-9d14-f7206bd8a824\") " pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" Jul 03 14:44:10.632817 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:10.632777 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" Jul 03 14:44:10.772778 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:10.772752 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg"] Jul 03 14:44:10.774776 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:44:10.774736 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf94e282_da48_4322_9d14_f7206bd8a824.slice/crio-812f65c284a576ca540727ef697276871f999bfc6abd76f0b8289da3dd0a2fc2 WatchSource:0}: Error finding container 812f65c284a576ca540727ef697276871f999bfc6abd76f0b8289da3dd0a2fc2: Status 404 returned error can't find the container with id 812f65c284a576ca540727ef697276871f999bfc6abd76f0b8289da3dd0a2fc2 Jul 03 14:44:11.017641 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:11.017553 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" event={"ID":"af94e282-da48-4322-9d14-f7206bd8a824","Type":"ContainerStarted","Data":"812f65c284a576ca540727ef697276871f999bfc6abd76f0b8289da3dd0a2fc2"} Jul 03 14:44:12.023679 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:12.023617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" event={"ID":"af94e282-da48-4322-9d14-f7206bd8a824","Type":"ContainerStarted","Data":"f21bc8ac37f2fede941339d9a268046eea239e7edb0b2cf1f742857760a8501d"} Jul 03 14:44:12.024060 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:12.023697 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" Jul 03 14:44:12.041389 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:12.041324 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" podStartSLOduration=2.710090408 podStartE2EDuration="3.041305042s" totalImagesPullingTime="331.214634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 14:44:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:44:12.040806283 +0000 UTC m=+2130.397090902" watchObservedRunningTime="2026-07-03 14:44:12.041305042 +0000 UTC m=+2130.397589663" Jul 03 14:44:18.033878 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:44:18.033844 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" Jul 03 14:48:42.275467 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:48:42.275440 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:48:42.278644 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:48:42.278615 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:48:42.278959 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:48:42.278942 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:48:42.282129 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:48:42.282109 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vtmd7_c5f6cf69-3f8b-4afe-a474-4dffa0448079/ovn-acl-logging/0.log" Jul 03 14:51:10.485982 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:10.485946 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x"] Jul 03 14:51:10.486785 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:10.486741 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" podUID="19c702a4-76c3-49a8-b712-205767fb717c" containerName="maas-api" containerID="cri-o://2e4941d098420cc3f6c618948f3919ff6aef471b2ec5aebcbf3d753fc3bce6c9" gracePeriod=30 Jul 03 14:51:11.735007 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:11.734974 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" Jul 03 14:51:11.905009 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:11.904907 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c702a4-76c3-49a8-b712-205767fb717c-maas-api-tls\") pod \"19c702a4-76c3-49a8-b712-205767fb717c\" (UID: \"19c702a4-76c3-49a8-b712-205767fb717c\") " Jul 03 14:51:11.905009 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:11.904960 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q54z\" (UniqueName: \"kubernetes.io/projected/19c702a4-76c3-49a8-b712-205767fb717c-kube-api-access-8q54z\") pod \"19c702a4-76c3-49a8-b712-205767fb717c\" (UID: \"19c702a4-76c3-49a8-b712-205767fb717c\") " Jul 03 14:51:11.905216 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:11.905093 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c702a4-76c3-49a8-b712-205767fb717c-openshift-service-ca\") pod \"19c702a4-76c3-49a8-b712-205767fb717c\" (UID: \"19c702a4-76c3-49a8-b712-205767fb717c\") " Jul 03 14:51:11.905467 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:11.905437 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c702a4-76c3-49a8-b712-205767fb717c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "19c702a4-76c3-49a8-b712-205767fb717c" (UID: "19c702a4-76c3-49a8-b712-205767fb717c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:51:11.907275 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:11.907257 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c702a4-76c3-49a8-b712-205767fb717c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "19c702a4-76c3-49a8-b712-205767fb717c" (UID: "19c702a4-76c3-49a8-b712-205767fb717c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 14:51:11.907363 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:11.907342 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c702a4-76c3-49a8-b712-205767fb717c-kube-api-access-8q54z" (OuterVolumeSpecName: "kube-api-access-8q54z") pod "19c702a4-76c3-49a8-b712-205767fb717c" (UID: "19c702a4-76c3-49a8-b712-205767fb717c"). InnerVolumeSpecName "kube-api-access-8q54z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:51:12.005905 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:12.005874 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c702a4-76c3-49a8-b712-205767fb717c-maas-api-tls\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:51:12.005905 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:12.005900 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q54z\" (UniqueName: \"kubernetes.io/projected/19c702a4-76c3-49a8-b712-205767fb717c-kube-api-access-8q54z\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:51:12.005905 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:12.005910 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c702a4-76c3-49a8-b712-205767fb717c-openshift-service-ca\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:51:12.006401 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:12.006378 2576 generic.go:356] "Generic (PLEG): container finished" podID="19c702a4-76c3-49a8-b712-205767fb717c" containerID="2e4941d098420cc3f6c618948f3919ff6aef471b2ec5aebcbf3d753fc3bce6c9" exitCode=0 Jul 03 14:51:12.006454 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:12.006427 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" Jul 03 14:51:12.006454 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:12.006436 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" event={"ID":"19c702a4-76c3-49a8-b712-205767fb717c","Type":"ContainerDied","Data":"2e4941d098420cc3f6c618948f3919ff6aef471b2ec5aebcbf3d753fc3bce6c9"} Jul 03 14:51:12.006533 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:12.006480 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x" event={"ID":"19c702a4-76c3-49a8-b712-205767fb717c","Type":"ContainerDied","Data":"8c56d4c97771ce13c018decae65329575fd21c09718f717cd5b527ecfdaf7064"} Jul 03 14:51:12.006533 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:12.006510 2576 scope.go:117] "RemoveContainer" containerID="2e4941d098420cc3f6c618948f3919ff6aef471b2ec5aebcbf3d753fc3bce6c9" Jul 03 14:51:12.015640 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:12.015620 2576 scope.go:117] "RemoveContainer" containerID="2e4941d098420cc3f6c618948f3919ff6aef471b2ec5aebcbf3d753fc3bce6c9" Jul 03 14:51:12.015936 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:51:12.015918 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e4941d098420cc3f6c618948f3919ff6aef471b2ec5aebcbf3d753fc3bce6c9\": container with ID starting with 2e4941d098420cc3f6c618948f3919ff6aef471b2ec5aebcbf3d753fc3bce6c9 not found: ID does not exist" containerID="2e4941d098420cc3f6c618948f3919ff6aef471b2ec5aebcbf3d753fc3bce6c9" Jul 03 14:51:12.015995 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:12.015943 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e4941d098420cc3f6c618948f3919ff6aef471b2ec5aebcbf3d753fc3bce6c9"} err="failed to get container status \"2e4941d098420cc3f6c618948f3919ff6aef471b2ec5aebcbf3d753fc3bce6c9\": rpc error: code = NotFound desc = could not find container \"2e4941d098420cc3f6c618948f3919ff6aef471b2ec5aebcbf3d753fc3bce6c9\": container with ID starting with 2e4941d098420cc3f6c618948f3919ff6aef471b2ec5aebcbf3d753fc3bce6c9 not found: ID does not exist" Jul 03 14:51:12.031509 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:12.031478 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x"] Jul 03 14:51:12.036864 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:12.036838 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-5f1e04-fcf695f6f-9qf4x"] Jul 03 14:51:12.167497 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:12.167419 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19c702a4-76c3-49a8-b712-205767fb717c" path="/var/lib/kubelet/pods/19c702a4-76c3-49a8-b712-205767fb717c/volumes" Jul 03 14:51:17.743945 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:17.743900 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg"] Jul 03 14:51:17.744337 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:17.744183 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" podUID="af94e282-da48-4322-9d14-f7206bd8a824" containerName="maas-api" containerID="cri-o://f21bc8ac37f2fede941339d9a268046eea239e7edb0b2cf1f742857760a8501d" gracePeriod=30 Jul 03 14:51:17.992447 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:17.992422 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" Jul 03 14:51:18.039609 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:18.039577 2576 generic.go:356] "Generic (PLEG): container finished" podID="af94e282-da48-4322-9d14-f7206bd8a824" containerID="f21bc8ac37f2fede941339d9a268046eea239e7edb0b2cf1f742857760a8501d" exitCode=0 Jul 03 14:51:18.039804 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:18.039611 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" event={"ID":"af94e282-da48-4322-9d14-f7206bd8a824","Type":"ContainerDied","Data":"f21bc8ac37f2fede941339d9a268046eea239e7edb0b2cf1f742857760a8501d"} Jul 03 14:51:18.039804 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:18.039642 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" Jul 03 14:51:18.039804 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:18.039693 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg" event={"ID":"af94e282-da48-4322-9d14-f7206bd8a824","Type":"ContainerDied","Data":"812f65c284a576ca540727ef697276871f999bfc6abd76f0b8289da3dd0a2fc2"} Jul 03 14:51:18.039804 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:18.039716 2576 scope.go:117] "RemoveContainer" containerID="f21bc8ac37f2fede941339d9a268046eea239e7edb0b2cf1f742857760a8501d" Jul 03 14:51:18.048244 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:18.048228 2576 scope.go:117] "RemoveContainer" containerID="f21bc8ac37f2fede941339d9a268046eea239e7edb0b2cf1f742857760a8501d" Jul 03 14:51:18.048549 ip-10-0-140-82 kubenswrapper[2576]: E0703 14:51:18.048530 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f21bc8ac37f2fede941339d9a268046eea239e7edb0b2cf1f742857760a8501d\": container with ID starting with f21bc8ac37f2fede941339d9a268046eea239e7edb0b2cf1f742857760a8501d not found: ID does not exist" containerID="f21bc8ac37f2fede941339d9a268046eea239e7edb0b2cf1f742857760a8501d" Jul 03 14:51:18.048603 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:18.048558 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21bc8ac37f2fede941339d9a268046eea239e7edb0b2cf1f742857760a8501d"} err="failed to get container status \"f21bc8ac37f2fede941339d9a268046eea239e7edb0b2cf1f742857760a8501d\": rpc error: code = NotFound desc = could not find container \"f21bc8ac37f2fede941339d9a268046eea239e7edb0b2cf1f742857760a8501d\": container with ID starting with f21bc8ac37f2fede941339d9a268046eea239e7edb0b2cf1f742857760a8501d not found: ID does not exist" Jul 03 14:51:18.056927 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:18.056907 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af94e282-da48-4322-9d14-f7206bd8a824-maas-api-tls\") pod \"af94e282-da48-4322-9d14-f7206bd8a824\" (UID: \"af94e282-da48-4322-9d14-f7206bd8a824\") " Jul 03 14:51:18.057033 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:18.056941 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkq65\" (UniqueName: \"kubernetes.io/projected/af94e282-da48-4322-9d14-f7206bd8a824-kube-api-access-nkq65\") pod \"af94e282-da48-4322-9d14-f7206bd8a824\" (UID: \"af94e282-da48-4322-9d14-f7206bd8a824\") " Jul 03 14:51:18.057033 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:18.056976 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af94e282-da48-4322-9d14-f7206bd8a824-openshift-service-ca\") pod \"af94e282-da48-4322-9d14-f7206bd8a824\" (UID: \"af94e282-da48-4322-9d14-f7206bd8a824\") " Jul 03 14:51:18.057352 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:18.057328 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af94e282-da48-4322-9d14-f7206bd8a824-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "af94e282-da48-4322-9d14-f7206bd8a824" (UID: "af94e282-da48-4322-9d14-f7206bd8a824"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 14:51:18.059018 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:18.058999 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af94e282-da48-4322-9d14-f7206bd8a824-kube-api-access-nkq65" (OuterVolumeSpecName: "kube-api-access-nkq65") pod "af94e282-da48-4322-9d14-f7206bd8a824" (UID: "af94e282-da48-4322-9d14-f7206bd8a824"). InnerVolumeSpecName "kube-api-access-nkq65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 14:51:18.059253 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:18.059234 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af94e282-da48-4322-9d14-f7206bd8a824-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "af94e282-da48-4322-9d14-f7206bd8a824" (UID: "af94e282-da48-4322-9d14-f7206bd8a824"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 14:51:18.157519 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:18.157484 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af94e282-da48-4322-9d14-f7206bd8a824-openshift-service-ca\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:51:18.157519 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:18.157514 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af94e282-da48-4322-9d14-f7206bd8a824-maas-api-tls\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:51:18.157519 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:18.157523 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkq65\" (UniqueName: \"kubernetes.io/projected/af94e282-da48-4322-9d14-f7206bd8a824-kube-api-access-nkq65\") on node \"ip-10-0-140-82.ec2.internal\" DevicePath \"\"" Jul 03 14:51:18.355580 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:18.355504 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg"] Jul 03 14:51:18.357217 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:18.357197 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-28686a-656dd999c6-2nlhg"] Jul 03 14:51:20.167958 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:20.167930 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af94e282-da48-4322-9d14-f7206bd8a824" path="/var/lib/kubelet/pods/af94e282-da48-4322-9d14-f7206bd8a824/volumes" Jul 03 14:51:29.116176 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:29.116139 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-5b46495cf4-rhj5r_4bb76db1-1a34-47dc-9eb5-8624e53dd4e5/maas-api/0.log" Jul 03 14:51:29.459460 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:29.459365 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bfc956fc6-5wsvb_1e117128-6ed2-4f63-a796-7d4af29ddb63/manager/0.log" Jul 03 14:51:29.799587 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:29.799562 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-977f7bbf6-7cn9c_11ed09d0-3eca-431e-8f56-62433bc47804/copy-manifests/0.log" Jul 03 14:51:29.818080 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:29.818039 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-977f7bbf6-7cn9c_11ed09d0-3eca-431e-8f56-62433bc47804/manager/0.log" Jul 03 14:51:32.628979 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:32.628945 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x8sgn_c01fd94e-5c98-4c4e-a984-9e21f891425b/istio-proxy/0.log" Jul 03 14:51:32.755813 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:32.755772 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-q6mtp_c49627e5-6591-4449-9ffd-7016ac083996/discovery/0.log" Jul 03 14:51:33.328993 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:33.328960 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-w69bs_0bf13177-f69d-4de3-8f4f-93da301ec70a/payload-processing/0.log" Jul 03 14:51:33.435795 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:33.435765 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-578dddfc57-dnhg6_18604274-740f-4f0b-bc1c-fb45e29fdc12/router/0.log" Jul 03 14:51:33.877999 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:33.877974 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-vjvbz_1ab82996-7cc1-431f-97fb-c7435fcb1d31/main/0.log" Jul 03 14:51:34.205946 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:34.205867 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-rwf6s_bca895b8-a0a9-4211-86ba-8b5c956d1552/main/0.log" Jul 03 14:51:40.845394 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:40.845354 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xn7tc_21624337-43ef-4f45-8118-63b9de2f9ed9/global-pull-secret-syncer/0.log" Jul 03 14:51:40.939107 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:40.939054 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lbt8h_99aa3d88-a61c-4e92-adc3-a69b48070430/konnectivity-agent/0.log" Jul 03 14:51:41.021146 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:41.021114 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-82.ec2.internal_aabef1e1070cbce5457cd456994a35fb/haproxy/0.log" Jul 03 14:51:48.663457 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:48.663421 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-b4gdz_87429e78-2975-4107-8cd7-aebcd695ad4f/kube-state-metrics/0.log" Jul 03 14:51:48.683408 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:48.683382 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-b4gdz_87429e78-2975-4107-8cd7-aebcd695ad4f/kube-rbac-proxy-main/0.log" Jul 03 14:51:48.704103 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:48.704053 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-b4gdz_87429e78-2975-4107-8cd7-aebcd695ad4f/kube-rbac-proxy-self/0.log" Jul 03 14:51:48.818093 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:48.818060 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-97nhz_b5d47a63-27a9-4aa2-9e30-0c031a2da741/node-exporter/0.log" Jul 03 14:51:48.838202 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:48.838177 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-97nhz_b5d47a63-27a9-4aa2-9e30-0c031a2da741/kube-rbac-proxy/0.log" Jul 03 14:51:48.865037 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:48.865013 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-97nhz_b5d47a63-27a9-4aa2-9e30-0c031a2da741/init-textfile/0.log" Jul 03 14:51:49.297447 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:49.297417 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-tslg9_398622f5-0feb-4a05-a51d-64ff760605ca/prometheus-operator/0.log" Jul 03 14:51:49.312930 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:49.312899 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-tslg9_398622f5-0feb-4a05-a51d-64ff760605ca/kube-rbac-proxy/0.log" Jul 03 14:51:51.118945 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.118912 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8"] Jul 03 14:51:51.119351 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.119238 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c702a4-76c3-49a8-b712-205767fb717c" containerName="maas-api" Jul 03 14:51:51.119351 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.119249 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c702a4-76c3-49a8-b712-205767fb717c" containerName="maas-api" Jul 03 14:51:51.119351 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.119269 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af94e282-da48-4322-9d14-f7206bd8a824" containerName="maas-api" Jul 03 14:51:51.119351 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.119274 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="af94e282-da48-4322-9d14-f7206bd8a824" containerName="maas-api" Jul 03 14:51:51.119585 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.119371 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="19c702a4-76c3-49a8-b712-205767fb717c" containerName="maas-api" Jul 03 14:51:51.119585 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.119391 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="af94e282-da48-4322-9d14-f7206bd8a824" containerName="maas-api" Jul 03 14:51:51.122387 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.122368 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" Jul 03 14:51:51.125371 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.125350 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s2kmc\"/\"openshift-service-ca.crt\"" Jul 03 14:51:51.125488 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.125414 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s2kmc\"/\"kube-root-ca.crt\"" Jul 03 14:51:51.126542 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.126525 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s2kmc\"/\"default-dockercfg-fgmwv\"" Jul 03 14:51:51.132620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.132593 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8"] Jul 03 14:51:51.240921 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.240889 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1ee172e2-ada7-4aac-b02d-1ce87ab399c9-proc\") pod \"perf-node-gather-daemonset-dghs8\" (UID: \"1ee172e2-ada7-4aac-b02d-1ce87ab399c9\") " pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" Jul 03 14:51:51.240921 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.240927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ee172e2-ada7-4aac-b02d-1ce87ab399c9-sys\") pod \"perf-node-gather-daemonset-dghs8\" (UID: \"1ee172e2-ada7-4aac-b02d-1ce87ab399c9\") " pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" Jul 03 14:51:51.241135 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.240973 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1ee172e2-ada7-4aac-b02d-1ce87ab399c9-podres\") pod \"perf-node-gather-daemonset-dghs8\" (UID: \"1ee172e2-ada7-4aac-b02d-1ce87ab399c9\") " pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" Jul 03 14:51:51.241135 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.241014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qpqb\" (UniqueName: \"kubernetes.io/projected/1ee172e2-ada7-4aac-b02d-1ce87ab399c9-kube-api-access-7qpqb\") pod \"perf-node-gather-daemonset-dghs8\" (UID: \"1ee172e2-ada7-4aac-b02d-1ce87ab399c9\") " pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" Jul 03 14:51:51.241135 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.241032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ee172e2-ada7-4aac-b02d-1ce87ab399c9-lib-modules\") pod \"perf-node-gather-daemonset-dghs8\" (UID: \"1ee172e2-ada7-4aac-b02d-1ce87ab399c9\") " pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" Jul 03 14:51:51.342431 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.342397 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1ee172e2-ada7-4aac-b02d-1ce87ab399c9-podres\") pod \"perf-node-gather-daemonset-dghs8\" (UID: \"1ee172e2-ada7-4aac-b02d-1ce87ab399c9\") " pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" Jul 03 14:51:51.342617 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.342444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qpqb\" (UniqueName: \"kubernetes.io/projected/1ee172e2-ada7-4aac-b02d-1ce87ab399c9-kube-api-access-7qpqb\") pod \"perf-node-gather-daemonset-dghs8\" (UID: \"1ee172e2-ada7-4aac-b02d-1ce87ab399c9\") " pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" Jul 03 14:51:51.342617 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.342464 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ee172e2-ada7-4aac-b02d-1ce87ab399c9-lib-modules\") pod \"perf-node-gather-daemonset-dghs8\" (UID: \"1ee172e2-ada7-4aac-b02d-1ce87ab399c9\") " pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" Jul 03 14:51:51.342617 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.342507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1ee172e2-ada7-4aac-b02d-1ce87ab399c9-proc\") pod \"perf-node-gather-daemonset-dghs8\" (UID: \"1ee172e2-ada7-4aac-b02d-1ce87ab399c9\") " pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" Jul 03 14:51:51.342617 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.342523 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ee172e2-ada7-4aac-b02d-1ce87ab399c9-sys\") pod \"perf-node-gather-daemonset-dghs8\" (UID: \"1ee172e2-ada7-4aac-b02d-1ce87ab399c9\") " pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" Jul 03 14:51:51.342617 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.342590 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1ee172e2-ada7-4aac-b02d-1ce87ab399c9-podres\") pod \"perf-node-gather-daemonset-dghs8\" (UID: \"1ee172e2-ada7-4aac-b02d-1ce87ab399c9\") " pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" Jul 03 14:51:51.342617 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.342609 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ee172e2-ada7-4aac-b02d-1ce87ab399c9-sys\") pod \"perf-node-gather-daemonset-dghs8\" (UID: \"1ee172e2-ada7-4aac-b02d-1ce87ab399c9\") " pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" Jul 03 14:51:51.342905 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.342635 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1ee172e2-ada7-4aac-b02d-1ce87ab399c9-proc\") pod \"perf-node-gather-daemonset-dghs8\" (UID: \"1ee172e2-ada7-4aac-b02d-1ce87ab399c9\") " pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" Jul 03 14:51:51.342905 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.342715 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ee172e2-ada7-4aac-b02d-1ce87ab399c9-lib-modules\") pod \"perf-node-gather-daemonset-dghs8\" (UID: \"1ee172e2-ada7-4aac-b02d-1ce87ab399c9\") " pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" Jul 03 14:51:51.348079 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.348047 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/1.log" Jul 03 14:51:51.351614 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.351589 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qpqb\" (UniqueName: \"kubernetes.io/projected/1ee172e2-ada7-4aac-b02d-1ce87ab399c9-kube-api-access-7qpqb\") pod \"perf-node-gather-daemonset-dghs8\" (UID: \"1ee172e2-ada7-4aac-b02d-1ce87ab399c9\") " pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" Jul 03 14:51:51.358593 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.358566 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-6fj84_cdde4f72-8103-49d7-b324-366692ef7c99/console-operator/2.log" Jul 03 14:51:51.433122 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.433039 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" Jul 03 14:51:51.571708 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:51.571621 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8"] Jul 03 14:51:51.573872 ip-10-0-140-82 kubenswrapper[2576]: W0703 14:51:51.573845 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1ee172e2_ada7_4aac_b02d_1ce87ab399c9.slice/crio-3441c2bd3bb2f04ed589860eb09726849a4b550db24d9b3d3464943608c7b89e WatchSource:0}: Error finding container 3441c2bd3bb2f04ed589860eb09726849a4b550db24d9b3d3464943608c7b89e: Status 404 returned error can't find the container with id 3441c2bd3bb2f04ed589860eb09726849a4b550db24d9b3d3464943608c7b89e Jul 03 14:51:52.203454 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:52.203421 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" event={"ID":"1ee172e2-ada7-4aac-b02d-1ce87ab399c9","Type":"ContainerStarted","Data":"f149dbb1c27d323e47446e2313071f95d4a660e40172e938ce099f23ebd71f67"} Jul 03 14:51:52.203869 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:52.203473 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" event={"ID":"1ee172e2-ada7-4aac-b02d-1ce87ab399c9","Type":"ContainerStarted","Data":"3441c2bd3bb2f04ed589860eb09726849a4b550db24d9b3d3464943608c7b89e"} Jul 03 14:51:52.203869 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:52.203507 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" Jul 03 14:51:52.223346 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:52.223289 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" podStartSLOduration=1.223273254 podStartE2EDuration="1.223273254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 14:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 14:51:52.220376141 +0000 UTC m=+2590.576660763" watchObservedRunningTime="2026-07-03 14:51:52.223273254 +0000 UTC m=+2590.579557853" Jul 03 14:51:53.624175 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:53.624150 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wx4zj_2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4/dns/0.log" Jul 03 14:51:53.643064 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:53.643039 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wx4zj_2e0dda2d-eb99-478f-aca7-6f3b7f1a1dd4/kube-rbac-proxy/0.log" Jul 03 14:51:53.665242 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:53.665208 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4v7fc_928b4e88-1d06-41e9-99c9-efecc91c0ee9/dns-node-resolver/0.log" Jul 03 14:51:54.242211 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:54.242169 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d699c5c4-fv99h_d1e565a7-e9d7-4b4c-bc53-147e9014748a/registry/0.log" Jul 03 14:51:54.286385 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:54.286354 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hjwhk_b6e59bf8-184d-4593-aa5f-de1c5854ac77/node-ca/0.log" Jul 03 14:51:55.242169 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:55.242134 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x8sgn_c01fd94e-5c98-4c4e-a984-9e21f891425b/istio-proxy/0.log" Jul 03 14:51:55.435636 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:55.435605 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-q6mtp_c49627e5-6591-4449-9ffd-7016ac083996/discovery/0.log" Jul 03 14:51:55.672647 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:55.672611 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-w69bs_0bf13177-f69d-4de3-8f4f-93da301ec70a/payload-processing/0.log" Jul 03 14:51:55.694002 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:55.693966 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-578dddfc57-dnhg6_18604274-740f-4f0b-bc1c-fb45e29fdc12/router/0.log" Jul 03 14:51:56.409874 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:56.409839 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tn4pv_68e8c775-86ec-4944-9379-70d1bcfdbf69/serve-healthcheck-canary/0.log" Jul 03 14:51:56.967062 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:56.967031 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v5kvh_cc9da1fd-c331-406d-a767-5b50b53f696f/kube-rbac-proxy/0.log" Jul 03 14:51:56.986620 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:56.986579 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v5kvh_cc9da1fd-c331-406d-a767-5b50b53f696f/exporter/0.log" Jul 03 14:51:57.006888 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:57.006862 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v5kvh_cc9da1fd-c331-406d-a767-5b50b53f696f/extractor/0.log" Jul 03 14:51:58.218390 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:58.218360 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s2kmc/perf-node-gather-daemonset-dghs8" Jul 03 14:51:59.404395 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:59.404360 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-5b46495cf4-rhj5r_4bb76db1-1a34-47dc-9eb5-8624e53dd4e5/maas-api/0.log" Jul 03 14:51:59.597298 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:59.597257 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bfc956fc6-5wsvb_1e117128-6ed2-4f63-a796-7d4af29ddb63/manager/0.log" Jul 03 14:51:59.822317 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:59.822277 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-977f7bbf6-7cn9c_11ed09d0-3eca-431e-8f56-62433bc47804/manager/0.log" Jul 03 14:51:59.838963 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:51:59.838935 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-977f7bbf6-7cn9c_11ed09d0-3eca-431e-8f56-62433bc47804/copy-manifests/0.log" Jul 03 14:52:01.160589 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:52:01.160492 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x8sgn_c01fd94e-5c98-4c4e-a984-9e21f891425b/istio-proxy/0.log" Jul 03 14:52:01.349185 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:52:01.349133 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-q6mtp_c49627e5-6591-4449-9ffd-7016ac083996/discovery/0.log" Jul 03 14:52:01.512487 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:52:01.512400 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-w69bs_0bf13177-f69d-4de3-8f4f-93da301ec70a/payload-processing/0.log" Jul 03 14:52:01.527630 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:52:01.527588 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-578dddfc57-dnhg6_18604274-740f-4f0b-bc1c-fb45e29fdc12/router/0.log" Jul 03 14:52:01.559925 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:52:01.559900 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-78cbbf6689-fh8dn_7433f219-4c0d-45d2-b727-1ade4503e8b0/manager/0.log" Jul 03 14:52:02.802320 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:52:02.802176 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x8sgn_c01fd94e-5c98-4c4e-a984-9e21f891425b/istio-proxy/0.log" Jul 03 14:52:02.989200 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:52:02.989171 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-q6mtp_c49627e5-6591-4449-9ffd-7016ac083996/discovery/0.log" Jul 03 14:52:03.184163 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:52:03.184075 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-w69bs_0bf13177-f69d-4de3-8f4f-93da301ec70a/payload-processing/0.log" Jul 03 14:52:03.198347 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:52:03.198318 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-578dddfc57-dnhg6_18604274-740f-4f0b-bc1c-fb45e29fdc12/router/0.log" Jul 03 14:52:08.002287 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:52:08.002251 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7w6pw_c77a739f-276a-4602-824b-874f70e83982/kube-multus/0.log" Jul 03 14:52:08.172596 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:52:08.172567 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fvf88_f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d/kube-multus-additional-cni-plugins/0.log" Jul 03 14:52:08.192938 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:52:08.192910 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fvf88_f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d/egress-router-binary-copy/0.log" Jul 03 14:52:08.211962 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:52:08.211935 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fvf88_f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d/cni-plugins/0.log" Jul 03 14:52:08.233590 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:52:08.233562 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fvf88_f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d/bond-cni-plugin/0.log" Jul 03 14:52:08.252820 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:52:08.252732 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fvf88_f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d/routeoverride-cni/0.log" Jul 03 14:52:08.273809 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:52:08.273774 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fvf88_f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d/whereabouts-cni-bincopy/0.log" Jul 03 14:52:08.294167 ip-10-0-140-82 kubenswrapper[2576]: I0703 14:52:08.294136 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fvf88_f7d9de01-3ada-4cb0-ae41-49f2c3f70a5d/whereabouts-cni/0.log"