Jul 29 20:44:27.823601 ip-10-0-137-216 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 29 20:44:27.823745 ip-10-0-137-216 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 29 20:44:27.823844 ip-10-0-137-216 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 29 20:44:27.824365 ip-10-0-137-216 systemd[1]: Failed to start Kubernetes Kubelet. Jul 29 20:44:37.845354 ip-10-0-137-216 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 29 20:44:37.845368 ip-10-0-137-216 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 106bdbe09373457aba90179883675317 -- Jul 29 20:46:21.808592 ip-10-0-137-216 systemd[1]: Starting Kubernetes Kubelet... Jul 29 20:46:22.159511 ip-10-0-137-216 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 20:46:22.159511 ip-10-0-137-216 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 20:46:22.159511 ip-10-0-137-216 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 20:46:22.159511 ip-10-0-137-216 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 20:46:22.159511 ip-10-0-137-216 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 20:46:22.161672 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.161583 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 20:46:22.165549 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165532 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 20:46:22.165549 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165549 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165552 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165555 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165558 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165561 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165563 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165567 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165570 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165572 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165575 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165578 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165581 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165583 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165586 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165590 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165592 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165595 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165597 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165600 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165603 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 20:46:22.165613 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165606 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165609 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165612 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165615 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165618 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165621 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165624 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165627 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165630 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165633 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165636 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165639 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165641 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165644 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165647 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165651 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165654 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165658 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165661 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165664 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 20:46:22.166088 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165666 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165669 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165672 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165675 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165677 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165680 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165683 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165685 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165688 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165690 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165693 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165696 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165699 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165702 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165705 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165708 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165711 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165713 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165716 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165719 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 20:46:22.166648 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165721 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165724 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165726 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165731 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165735 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165738 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165741 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165744 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165747 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165750 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165753 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165756 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165758 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165761 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165764 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165766 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165769 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165771 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165774 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165777 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 20:46:22.167133 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165779 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 20:46:22.167657 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165783 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 20:46:22.167657 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165785 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 20:46:22.167657 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165788 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 20:46:22.167657 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165791 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 20:46:22.167657 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165794 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 20:46:22.167657 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165797 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 20:46:22.167657 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165800 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 20:46:22.167657 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165802 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 20:46:22.167657 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165805 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 20:46:22.167657 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165809 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 20:46:22.167657 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165812 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 20:46:22.167657 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165815 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 20:46:22.167657 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165819 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 20:46:22.167657 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165822 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 20:46:22.167657 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165824 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 20:46:22.167657 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165827 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 20:46:22.167657 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.165829 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 20:46:22.167657 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166552 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 20:46:22.167657 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166559 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166562 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166565 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166568 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166572 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166575 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166577 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166580 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166583 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166585 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166588 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166591 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166593 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166596 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166598 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166601 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166604 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166607 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166610 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166613 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 20:46:22.168117 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166616 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166618 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166621 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166624 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166627 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166629 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166632 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166635 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166637 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166640 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166643 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166645 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166648 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166651 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166654 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166656 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166659 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166663 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166666 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166669 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 20:46:22.168631 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166671 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166674 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166677 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166679 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166682 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166684 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166687 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166689 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166693 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166695 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166698 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166701 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166703 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166706 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166709 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166711 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166714 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166716 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166719 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166721 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 20:46:22.169123 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166724 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166726 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166729 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166733 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166735 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166738 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166741 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166745 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166749 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166752 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166755 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166758 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166761 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166764 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166767 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166769 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166774 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166777 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166781 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166784 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 20:46:22.169632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166787 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 20:46:22.170112 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166791 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 20:46:22.170112 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166795 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 20:46:22.170112 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166798 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 20:46:22.170112 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166801 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 20:46:22.170112 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166804 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 20:46:22.170112 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166806 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 20:46:22.170112 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166809 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 20:46:22.170112 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166812 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 20:46:22.170112 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166814 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 20:46:22.170112 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166817 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 20:46:22.170112 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166819 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 20:46:22.170112 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166822 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 20:46:22.170112 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166824 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 20:46:22.170112 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166827 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 20:46:22.170112 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166829 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 20:46:22.170112 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166832 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 20:46:22.170112 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.166835 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 20:46:22.170112 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167224 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 20:46:22.170112 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167229 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 20:46:22.170612 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167233 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 20:46:22.170612 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167236 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 20:46:22.170612 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167240 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 20:46:22.170612 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167243 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 20:46:22.170612 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167246 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 20:46:22.170612 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167248 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 20:46:22.170612 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167251 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 20:46:22.170612 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167253 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 20:46:22.170612 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167270 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 20:46:22.170612 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167273 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 20:46:22.170612 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167276 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 20:46:22.170612 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167279 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 20:46:22.170612 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167281 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 20:46:22.170612 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167285 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 20:46:22.170612 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167288 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 20:46:22.170612 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167292 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 20:46:22.170612 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167295 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 20:46:22.170612 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167298 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 20:46:22.170612 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167301 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167303 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167306 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167309 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167312 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167317 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167320 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167323 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167325 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167328 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167331 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167333 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167336 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167338 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167341 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167344 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167346 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167349 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167353 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167356 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 20:46:22.171069 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167358 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167361 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167364 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167367 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167370 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167373 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167377 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167380 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167383 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167386 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167389 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167392 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167395 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167397 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167400 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167403 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167405 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167410 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167412 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167415 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 20:46:22.171595 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167418 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167420 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167423 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167425 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167429 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167433 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167435 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167438 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167441 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167444 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167447 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167450 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167453 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167455 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167459 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167462 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167465 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167467 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167470 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167473 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 20:46:22.172132 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167475 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167478 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167480 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167483 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167486 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167489 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167492 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167495 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167497 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167501 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167504 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167507 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167509 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167512 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167514 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167517 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167519 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167523 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.167526 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167602 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 20:46:22.172632 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167608 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167615 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167619 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167624 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167627 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167631 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167636 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167639 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167642 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167645 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167648 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167652 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167655 2562 flags.go:64] FLAG: --cgroup-root="" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167658 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167661 2562 flags.go:64] FLAG: --client-ca-file="" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167664 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167667 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167672 2562 flags.go:64] FLAG: --cluster-domain="" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167675 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167679 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167682 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167686 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167690 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 20:46:22.173111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167693 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167696 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167700 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167703 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167707 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167710 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167714 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167717 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167726 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167729 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167732 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167735 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167738 2562 flags.go:64] FLAG: --enable-server="true" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167741 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167745 2562 flags.go:64] FLAG: --event-burst="100" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167748 2562 flags.go:64] FLAG: --event-qps="50" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167751 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167754 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167757 2562 flags.go:64] FLAG: --eviction-hard="" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167761 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167764 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167767 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167770 2562 flags.go:64] FLAG: --eviction-soft="" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167773 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167776 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 20:46:22.173748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167779 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167781 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167785 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167787 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167790 2562 flags.go:64] FLAG: --feature-gates="" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167796 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167799 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167802 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167805 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167808 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167811 2562 flags.go:64] FLAG: --help="false" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167814 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-137-216.ec2.internal" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167817 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167821 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167824 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167828 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167831 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167834 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167837 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167840 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167843 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167846 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167849 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167852 2562 flags.go:64] FLAG: --kube-reserved="" Jul 29 20:46:22.174397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167855 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167858 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167861 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167864 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167866 2562 flags.go:64] FLAG: --lock-file="" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167869 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167872 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167875 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167880 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167883 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167886 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167889 2562 flags.go:64] FLAG: --logging-format="text" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167892 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167897 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167900 2562 flags.go:64] FLAG: --manifest-url="" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167903 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167907 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167910 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167914 2562 flags.go:64] FLAG: --max-pods="110" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167917 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167920 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167924 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167927 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167930 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167933 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 20:46:22.174990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167936 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167943 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167946 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167949 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167952 2562 flags.go:64] FLAG: --pod-cidr="" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167955 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167960 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167963 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167966 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167969 2562 flags.go:64] FLAG: --port="10250" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167972 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167975 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e8441158dcbf97b9" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167978 2562 flags.go:64] FLAG: --qos-reserved="" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167981 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167984 2562 flags.go:64] FLAG: --register-node="true" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167987 2562 flags.go:64] FLAG: --register-with-taints="" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167991 2562 flags.go:64] FLAG: --registry-burst="10" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167993 2562 flags.go:64] FLAG: --registry-qps="5" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.167996 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168000 2562 flags.go:64] FLAG: --reserved-memory="" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168004 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168007 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168010 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168013 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168016 2562 flags.go:64] FLAG: --runonce="false" Jul 29 20:46:22.175616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168019 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168022 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168025 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168028 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168032 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168035 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168038 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168041 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168044 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168047 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168049 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168052 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168055 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168059 2562 flags.go:64] FLAG: --system-cgroups="" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168062 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168067 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168070 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168073 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168077 2562 flags.go:64] FLAG: --tls-min-version="" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168080 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168083 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168086 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168088 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168091 2562 flags.go:64] FLAG: --v="2" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168095 2562 flags.go:64] FLAG: --version="false" Jul 29 20:46:22.176241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168099 2562 flags.go:64] FLAG: --vmodule="" Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168103 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168108 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168222 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168226 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168229 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168232 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168235 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168237 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168240 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168243 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168247 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168250 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168253 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168267 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168270 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168273 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168275 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168278 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168281 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 20:46:22.176961 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168284 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168287 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168290 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168292 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168295 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168298 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168300 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168303 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168306 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168309 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168311 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168314 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168317 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168319 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168323 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168326 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168329 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168332 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168334 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168338 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 20:46:22.177512 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168341 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 20:46:22.178008 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168345 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 20:46:22.178008 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168347 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 20:46:22.178008 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168352 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 20:46:22.178008 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168354 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 20:46:22.178008 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168357 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 20:46:22.178008 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168359 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 20:46:22.178008 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168362 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 20:46:22.178008 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168365 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 20:46:22.178008 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168368 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 20:46:22.178008 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168370 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 20:46:22.178008 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168373 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 20:46:22.178008 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168376 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 20:46:22.178008 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168378 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 20:46:22.178008 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168381 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 20:46:22.178008 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168384 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 20:46:22.178008 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168386 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 20:46:22.178008 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168389 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 20:46:22.178008 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168392 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 20:46:22.178008 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168394 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 20:46:22.178486 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168397 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 20:46:22.178486 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168399 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 20:46:22.178486 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168402 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 20:46:22.178486 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168406 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 20:46:22.178486 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168409 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 20:46:22.178486 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168412 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 20:46:22.178486 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168416 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 20:46:22.178486 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168419 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 20:46:22.178486 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168422 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 20:46:22.178486 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168424 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 20:46:22.178486 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168427 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 20:46:22.178486 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168430 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 20:46:22.178486 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168432 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 20:46:22.178486 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168435 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 20:46:22.178486 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168438 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 20:46:22.178486 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168442 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 20:46:22.178486 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168445 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 20:46:22.178486 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168447 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 20:46:22.178486 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168450 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 20:46:22.178953 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168453 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 20:46:22.178953 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168455 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 20:46:22.178953 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168458 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 20:46:22.178953 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168461 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 20:46:22.178953 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168464 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 20:46:22.178953 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168466 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 20:46:22.178953 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168469 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 20:46:22.178953 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168471 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 20:46:22.178953 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168474 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 20:46:22.178953 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168477 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 20:46:22.178953 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168480 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 20:46:22.178953 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168483 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 20:46:22.178953 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168486 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 20:46:22.178953 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168490 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 20:46:22.178953 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168493 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 20:46:22.178953 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168496 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 20:46:22.178953 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168499 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 20:46:22.178953 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168502 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 20:46:22.178953 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168505 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 20:46:22.179425 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168509 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 20:46:22.179425 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168511 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 20:46:22.179425 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168514 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 20:46:22.179425 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.168517 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 20:46:22.179425 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.168526 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 20:46:22.179425 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.175111 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 20:46:22.179425 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.175129 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 20:46:22.179425 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.175149 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 20:46:22.179425 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.175154 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 20:46:22.179425 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175212 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 20:46:22.179425 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175217 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 20:46:22.179425 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175222 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 20:46:22.179425 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175225 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 20:46:22.179425 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175229 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 20:46:22.179425 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175232 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 20:46:22.179791 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175234 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 20:46:22.179791 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175237 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 20:46:22.179791 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175240 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 20:46:22.179791 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175243 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 20:46:22.179791 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175246 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 20:46:22.179791 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175249 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 20:46:22.179791 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175253 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 20:46:22.179791 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175271 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 20:46:22.179791 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175276 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 20:46:22.179791 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175281 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 20:46:22.179791 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175284 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 20:46:22.179791 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175287 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 20:46:22.179791 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175290 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 20:46:22.179791 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175293 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 20:46:22.179791 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175297 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 20:46:22.179791 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175299 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 20:46:22.179791 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175302 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 20:46:22.179791 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175305 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 20:46:22.179791 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175308 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 20:46:22.180269 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175310 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 20:46:22.180269 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175313 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 20:46:22.180269 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175316 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 20:46:22.180269 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175318 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 20:46:22.180269 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175322 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 20:46:22.180269 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175325 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 20:46:22.180269 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175328 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 20:46:22.180269 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175331 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 20:46:22.180269 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175334 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 20:46:22.180269 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175336 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 20:46:22.180269 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175339 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 20:46:22.180269 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175342 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 20:46:22.180269 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175344 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 20:46:22.180269 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175347 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 20:46:22.180269 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175350 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 20:46:22.180269 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175353 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 20:46:22.180269 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175355 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 20:46:22.180269 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175358 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 20:46:22.180269 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175361 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 20:46:22.180733 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175363 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 20:46:22.180733 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175367 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 20:46:22.180733 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175369 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 20:46:22.180733 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175372 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 20:46:22.180733 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175376 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 20:46:22.180733 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175380 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 20:46:22.180733 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175383 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 20:46:22.180733 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175385 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 20:46:22.180733 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175388 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 20:46:22.180733 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175391 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 20:46:22.180733 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175394 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 20:46:22.180733 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175396 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 20:46:22.180733 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175399 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 20:46:22.180733 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175401 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 20:46:22.180733 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175404 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 20:46:22.180733 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175406 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 20:46:22.180733 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175409 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 20:46:22.180733 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175412 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 20:46:22.180733 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175415 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 20:46:22.181191 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175419 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 20:46:22.181191 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175422 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 20:46:22.181191 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175425 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 20:46:22.181191 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175428 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 29 20:46:22.181191 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175431 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 20:46:22.181191 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175434 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 20:46:22.181191 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175436 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 20:46:22.181191 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175439 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 20:46:22.181191 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175442 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 20:46:22.181191 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175444 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 20:46:22.181191 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175447 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 20:46:22.181191 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175450 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 20:46:22.181191 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175452 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 20:46:22.181191 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175455 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 20:46:22.181191 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175458 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 20:46:22.181191 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175460 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 20:46:22.181191 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175463 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 20:46:22.181191 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175465 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 20:46:22.181191 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175468 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 20:46:22.181771 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175471 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 20:46:22.181771 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175473 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 20:46:22.181771 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175476 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 20:46:22.181771 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175478 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 20:46:22.181771 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175481 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 20:46:22.181771 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175484 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 20:46:22.181771 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175486 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 20:46:22.181771 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175488 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 20:46:22.181771 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175491 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 20:46:22.181771 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175494 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 20:46:22.181771 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175496 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 20:46:22.181771 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175499 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 20:46:22.181771 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175502 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 20:46:22.181771 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175505 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 20:46:22.181771 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175507 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 20:46:22.181771 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175510 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 20:46:22.181771 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175512 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 20:46:22.182454 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.175518 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 20:46:22.182454 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175633 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 20:46:22.182454 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175638 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 20:46:22.182454 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175641 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 20:46:22.182454 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175644 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 20:46:22.182454 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175647 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 29 20:46:22.182454 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175650 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 20:46:22.182454 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175653 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 20:46:22.182454 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175656 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 20:46:22.182454 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175659 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 20:46:22.182454 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175661 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 20:46:22.182454 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175664 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 20:46:22.182454 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175667 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 20:46:22.182454 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175670 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 20:46:22.182454 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175673 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175675 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175678 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175680 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175683 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175685 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175688 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175690 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175693 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175696 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175698 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175701 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175703 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175706 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175709 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175713 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175716 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175719 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175722 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175725 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 20:46:22.183072 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175727 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175730 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175732 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175735 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175737 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175740 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175742 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175745 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175748 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175750 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175753 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175756 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175759 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175762 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175764 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175767 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175769 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175772 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175774 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175777 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 20:46:22.183616 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175779 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 20:46:22.184102 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175782 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 20:46:22.184102 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175785 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 20:46:22.184102 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175787 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 20:46:22.184102 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175790 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 20:46:22.184102 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175792 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 20:46:22.184102 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175795 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 20:46:22.184102 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175797 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 20:46:22.184102 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175800 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 20:46:22.184102 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175803 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 20:46:22.184102 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175805 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 20:46:22.184102 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175808 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 20:46:22.184102 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175810 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 20:46:22.184102 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175813 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 20:46:22.184102 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175815 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 20:46:22.184102 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175818 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 20:46:22.184102 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175820 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 20:46:22.184102 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175823 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 20:46:22.184102 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175825 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 20:46:22.184102 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175828 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 20:46:22.184572 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175830 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 20:46:22.184572 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175833 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 20:46:22.184572 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175836 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 20:46:22.184572 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175838 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 20:46:22.184572 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175841 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 20:46:22.184572 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175844 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 20:46:22.184572 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175846 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 20:46:22.184572 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175849 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 20:46:22.184572 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175852 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 20:46:22.184572 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175855 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 20:46:22.184572 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175858 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 20:46:22.184572 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175860 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 20:46:22.184572 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175863 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 20:46:22.184572 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175865 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 20:46:22.184572 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175868 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 20:46:22.184572 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175870 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 20:46:22.184572 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175873 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 20:46:22.184572 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175876 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 20:46:22.184572 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175880 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 20:46:22.185033 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175883 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 20:46:22.185033 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175885 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 20:46:22.185033 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175889 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 20:46:22.185033 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175891 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 20:46:22.185033 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175894 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 20:46:22.185033 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175897 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 20:46:22.185033 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:22.175900 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 20:46:22.185033 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.175905 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 20:46:22.185033 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.176002 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 20:46:22.185033 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.177970 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 20:46:22.185033 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.179348 2562 server.go:1019] "Starting client certificate rotation" Jul 29 20:46:22.185033 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.179449 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 20:46:22.185033 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.180066 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 20:46:22.185541 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.182513 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 20:46:22.187001 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.186984 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 20:46:22.194997 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.194980 2562 log.go:25] "Validated CRI v1 runtime API" Jul 29 20:46:22.198197 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.198183 2562 log.go:25] "Validated CRI v1 image API" Jul 29 20:46:22.199553 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.199539 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 20:46:22.201652 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.201635 2562 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 93ffd832-c8cb-4b3e-84f8-22c6dcb50300:/dev/nvme0n1p3 eac290be-113d-46ef-99c1-f032882c14f3:/dev/nvme0n1p4] Jul 29 20:46:22.201703 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.201653 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 20:46:22.207695 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.207589 2562 manager.go:216] Machine: {Timestamp:2026-07-29 20:46:22.206405024 +0000 UTC m=+0.292168255 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106246 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24a72b80d2ce6cfc74a59276c7e636 SystemUUID:ec24a72b-80d2-ce6c-fc74-a59276c7e636 BootID:106bdbe0-9373-457a-ba90-179883675317 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:87:3e:16:88:79 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:87:3e:16:88:79 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:fb:1d:25:88:e7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 20:46:22.207695 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.207692 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 20:46:22.207804 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.207795 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 20:46:22.208069 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.208053 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 20:46:22.209969 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.209940 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 20:46:22.210108 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.209971 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-216.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 20:46:22.210944 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.210934 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 20:46:22.210977 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.210947 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 20:46:22.210977 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.210955 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 20:46:22.211411 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.211401 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 20:46:22.212512 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.212503 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 20:46:22.214908 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.214897 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 29 20:46:22.215014 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.215005 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 20:46:22.215056 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.215036 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 29 20:46:22.215056 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.215045 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 20:46:22.215122 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.215060 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 20:46:22.215122 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.215069 2562 kubelet.go:398] "Adding apiserver pod source" Jul 29 20:46:22.215122 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.215078 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 20:46:22.215464 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.215451 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 20:46:22.215499 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.215473 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 20:46:22.218318 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.218274 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 20:46:22.219965 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.219949 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 20:46:22.220053 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.219988 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 20:46:22.220535 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.220522 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 20:46:22.220596 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.220543 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 20:46:22.220596 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.220553 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 20:46:22.220596 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.220563 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 20:46:22.220596 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.220576 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 20:46:22.220596 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.220585 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 20:46:22.220798 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.220788 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 20:46:22.220845 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.220801 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 20:46:22.220845 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.220811 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 20:46:22.220845 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.220820 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 20:46:22.220845 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.220841 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 20:46:22.221008 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.220854 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 20:46:22.221008 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.220884 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 20:46:22.221008 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.220894 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 20:46:22.224006 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.223991 2562 server.go:1268] "Started kubelet" Jul 29 20:46:22.224129 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.224092 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 20:46:22.224430 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:22.224398 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 20:46:22.224518 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.224495 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-216.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 20:46:22.224595 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:22.224574 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-216.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 20:46:22.224595 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.224152 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 20:46:22.224693 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.224618 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 20:46:22.224883 ip-10-0-137-216 systemd[1]: Started Kubernetes Kubelet. Jul 29 20:46:22.226060 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.226045 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 20:46:22.227187 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.227170 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 29 20:46:22.229856 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:22.229817 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 20:46:22.229990 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:22.227790 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-216.ec2.internal.18c6dda0275e08e2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-216.ec2.internal,UID:ip-10-0-137-216.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-216.ec2.internal,},FirstTimestamp:2026-07-29 20:46:22.22396029 +0000 UTC m=+0.309723522,LastTimestamp:2026-07-29 20:46:22.22396029 +0000 UTC m=+0.309723522,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-216.ec2.internal,}" Jul 29 20:46:22.231979 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.231964 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 20:46:22.232057 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.232037 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 20:46:22.232118 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:22.232076 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-216.ec2.internal\" not found" Jul 29 20:46:22.232719 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.232679 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 20:46:22.232719 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.232681 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 20:46:22.232719 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.232719 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 20:46:22.232888 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.232816 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 29 20:46:22.232888 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.232823 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 29 20:46:22.233032 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.233018 2562 factory.go:55] Registering systemd factory Jul 29 20:46:22.233032 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.233033 2562 factory.go:223] Registration of the systemd container factory successfully Jul 29 20:46:22.233350 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.233330 2562 factory.go:153] Registering CRI-O factory Jul 29 20:46:22.233350 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.233350 2562 factory.go:223] Registration of the crio container factory successfully Jul 29 20:46:22.233493 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.233413 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 20:46:22.233493 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.233435 2562 factory.go:103] Registering Raw factory Jul 29 20:46:22.233493 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.233451 2562 manager.go:1196] Started watching for new ooms in manager Jul 29 20:46:22.233896 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.233881 2562 manager.go:349] Starting recovery of all containers Jul 29 20:46:22.234382 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:22.234356 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-216.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 29 20:46:22.234562 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:22.234539 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 29 20:46:22.246576 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.246559 2562 manager.go:354] Recovery completed Jul 29 20:46:22.250795 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.250781 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 20:46:22.252919 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.252902 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasSufficientMemory" Jul 29 20:46:22.252977 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.252932 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 20:46:22.252977 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.252942 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasSufficientPID" Jul 29 20:46:22.253512 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.253498 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 20:46:22.253512 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.253506 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 20:46:22.253605 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.253511 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 20:46:22.253605 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.253545 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 29 20:46:22.254148 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.254123 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasSufficientMemory" Jul 29 20:46:22.254190 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.254154 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 20:46:22.254190 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.254167 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasSufficientPID" Jul 29 20:46:22.254684 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.254663 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q6pwp" Jul 29 20:46:22.254862 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.254852 2562 policy_none.go:49] "None policy: Start" Jul 29 20:46:22.254891 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.254866 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 20:46:22.254891 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.254877 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 20:46:22.255794 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.255784 2562 policy_none.go:47] "Start" Jul 29 20:46:22.261147 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:22.261085 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-216.ec2.internal.18c6dda02917ea26 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-216.ec2.internal,UID:ip-10-0-137-216.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-216.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-216.ec2.internal,},FirstTimestamp:2026-07-29 20:46:22.252919334 +0000 UTC m=+0.338682569,LastTimestamp:2026-07-29 20:46:22.252919334 +0000 UTC m=+0.338682569,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-216.ec2.internal,}" Jul 29 20:46:22.261839 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.261826 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q6pwp" Jul 29 20:46:22.264890 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:22.264835 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-216.ec2.internal.18c6dda029182f2a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-216.ec2.internal,UID:ip-10-0-137-216.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-216.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-216.ec2.internal,},FirstTimestamp:2026-07-29 20:46:22.252937002 +0000 UTC m=+0.338700234,LastTimestamp:2026-07-29 20:46:22.252937002 +0000 UTC m=+0.338700234,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-216.ec2.internal,}" Jul 29 20:46:22.299002 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.298849 2562 manager.go:337] "Starting Device Plugin manager" Jul 29 20:46:22.299094 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:22.299026 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 20:46:22.299094 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.299040 2562 server.go:87] "Starting device plugin registration server" Jul 29 20:46:22.299295 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.299282 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 20:46:22.299337 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.299299 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 20:46:22.299419 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.299400 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 20:46:22.299512 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.299498 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 20:46:22.299512 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.299510 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 20:46:22.299988 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:22.299968 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 20:46:22.300047 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:22.300021 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-216.ec2.internal\" not found" Jul 29 20:46:22.334582 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.334552 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 20:46:22.335750 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.335737 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 20:46:22.335810 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.335786 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 20:46:22.335861 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.335812 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 20:46:22.335913 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:22.335859 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 20:46:22.338423 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.338406 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 20:46:22.399606 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.399587 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 20:46:22.400791 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.400775 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasSufficientMemory" Jul 29 20:46:22.400868 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.400807 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 20:46:22.400868 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.400817 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasSufficientPID" Jul 29 20:46:22.400868 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.400840 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-216.ec2.internal" Jul 29 20:46:22.410190 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.410147 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-216.ec2.internal" Jul 29 20:46:22.410190 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:22.410168 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-216.ec2.internal\": node \"ip-10-0-137-216.ec2.internal\" not found" Jul 29 20:46:22.434925 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:22.434901 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-216.ec2.internal\" not found" Jul 29 20:46:22.436594 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.436575 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-216.ec2.internal"] Jul 29 20:46:22.436662 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.436653 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 20:46:22.437473 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.437458 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasSufficientMemory" Jul 29 20:46:22.437557 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.437491 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 20:46:22.437557 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.437506 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasSufficientPID" Jul 29 20:46:22.438684 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.438669 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 20:46:22.438805 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.438791 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal" Jul 29 20:46:22.438850 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.438824 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 20:46:22.439318 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.439301 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasSufficientMemory" Jul 29 20:46:22.439407 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.439325 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 20:46:22.439407 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.439335 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasSufficientPID" Jul 29 20:46:22.439407 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.439399 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasSufficientMemory" Jul 29 20:46:22.439558 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.439428 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 20:46:22.439558 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.439442 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasSufficientPID" Jul 29 20:46:22.440528 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.440513 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-216.ec2.internal" Jul 29 20:46:22.440604 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.440545 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 20:46:22.441199 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.441182 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasSufficientMemory" Jul 29 20:46:22.441250 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.441214 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 20:46:22.441250 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:22.441229 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasSufficientPID" Jul 29 20:46:22.459907 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:22.459880 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-216.ec2.internal\" not found" node="ip-10-0-137-216.ec2.internal" Jul 29 20:46:22.463390 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:22.463372 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-216.ec2.internal\" not found" node="ip-10-0-137-216.ec2.internal" Jul 29 20:46:22.535866 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:22.535837 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-216.ec2.internal\" not found" Jul 29 20:46:23.848903 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:23.848870 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4cc20b7e84d827f5d2291ec9a66f9a73-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal\" (UID: \"4cc20b7e84d827f5d2291ec9a66f9a73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal" Jul 29 20:46:23.849031 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:23.848906 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cc20b7e84d827f5d2291ec9a66f9a73-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal\" (UID: \"4cc20b7e84d827f5d2291ec9a66f9a73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal" Jul 29 20:46:23.849031 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:23.848952 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4119ef05f02339ab6df3a952dfb37eab-config\") pod \"kube-apiserver-proxy-ip-10-0-137-216.ec2.internal\" (UID: \"4119ef05f02339ab6df3a952dfb37eab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-216.ec2.internal" Jul 29 20:46:23.850964 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:23.850944 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-216.ec2.internal\" not found" Jul 29 20:46:23.949708 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:23.949643 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4119ef05f02339ab6df3a952dfb37eab-config\") pod \"kube-apiserver-proxy-ip-10-0-137-216.ec2.internal\" (UID: \"4119ef05f02339ab6df3a952dfb37eab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-216.ec2.internal" Jul 29 20:46:23.949708 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:23.949682 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4cc20b7e84d827f5d2291ec9a66f9a73-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal\" (UID: \"4cc20b7e84d827f5d2291ec9a66f9a73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal" Jul 29 20:46:23.949827 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:23.949710 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cc20b7e84d827f5d2291ec9a66f9a73-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal\" (UID: \"4cc20b7e84d827f5d2291ec9a66f9a73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal" Jul 29 20:46:23.949827 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:23.949741 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cc20b7e84d827f5d2291ec9a66f9a73-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal\" (UID: \"4cc20b7e84d827f5d2291ec9a66f9a73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal" Jul 29 20:46:23.949827 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:23.949754 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4cc20b7e84d827f5d2291ec9a66f9a73-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal\" (UID: \"4cc20b7e84d827f5d2291ec9a66f9a73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal" Jul 29 20:46:23.949827 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:23.949740 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4119ef05f02339ab6df3a952dfb37eab-config\") pod \"kube-apiserver-proxy-ip-10-0-137-216.ec2.internal\" (UID: \"4119ef05f02339ab6df3a952dfb37eab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-216.ec2.internal" Jul 29 20:46:23.951723 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:23.951706 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-216.ec2.internal\" not found" Jul 29 20:46:23.976896 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:23.976874 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal" Jul 29 20:46:23.981378 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:23.981256 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-216.ec2.internal" Jul 29 20:46:24.052627 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:24.052575 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-216.ec2.internal\" not found" Jul 29 20:46:24.153089 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:24.153056 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-216.ec2.internal\" not found" Jul 29 20:46:24.253725 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:24.253640 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-216.ec2.internal\" not found" Jul 29 20:46:24.354184 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:24.354147 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-216.ec2.internal\" not found" Jul 29 20:46:24.393342 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:24.393313 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 20:46:24.393753 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:24.393458 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 20:46:24.455066 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:24.455034 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-216.ec2.internal\" not found" Jul 29 20:46:24.481188 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:24.481152 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 20:41:23 +0000 UTC" deadline="2028-03-19 05:40:15.474033738 +0000 UTC" Jul 29 20:46:24.481188 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:24.481184 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14360h53m50.992853419s" Jul 29 20:46:24.551607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:24.551582 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 20:46:24.554546 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:24.554522 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasSufficientMemory" Jul 29 20:46:24.554672 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:24.554559 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 20:46:24.554672 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:24.554575 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeHasSufficientPID" Jul 29 20:46:24.555729 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:24.555714 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-216.ec2.internal\" not found" Jul 29 20:46:24.614240 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:24.614206 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc20b7e84d827f5d2291ec9a66f9a73.slice/crio-ffb363ee62b045fb1a4ebf40a3ef583dbfbae5cd52be3c02af5dab8ac7169a10 WatchSource:0}: Error finding container ffb363ee62b045fb1a4ebf40a3ef583dbfbae5cd52be3c02af5dab8ac7169a10: Status 404 returned error can't find the container with id ffb363ee62b045fb1a4ebf40a3ef583dbfbae5cd52be3c02af5dab8ac7169a10 Jul 29 20:46:24.614582 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:24.614561 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4119ef05f02339ab6df3a952dfb37eab.slice/crio-052e60c7c445330f3940dbff5281c1fa2077974562bf941e5459e88c4aa47141 WatchSource:0}: Error finding container 052e60c7c445330f3940dbff5281c1fa2077974562bf941e5459e88c4aa47141: Status 404 returned error can't find the container with id 052e60c7c445330f3940dbff5281c1fa2077974562bf941e5459e88c4aa47141 Jul 29 20:46:24.618394 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:24.618379 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 20:46:24.655962 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:24.655932 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-216.ec2.internal\" not found" Jul 29 20:46:24.696731 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:24.696702 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 20:46:24.756557 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:24.756525 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-216.ec2.internal\" not found" Jul 29 20:46:24.837856 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:24.837787 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 20:46:24.847393 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:24.847366 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-216.ec2.internal" Jul 29 20:46:24.855650 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:24.855631 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 20:46:24.856491 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:24.856478 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal" Jul 29 20:46:24.866960 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:24.866932 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 20:46:24.986363 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:24.986330 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 20:46:25.430678 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.430646 2562 apiserver.go:52] "Watching apiserver" Jul 29 20:46:25.436199 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.436175 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 20:46:25.437194 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.437167 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-5tkhm","openshift-ovn-kubernetes/ovnkube-node-tplf2","kube-system/global-pull-secret-syncer-fhgt5","kube-system/konnectivity-agent-mc9jj","kube-system/kube-apiserver-proxy-ip-10-0-137-216.ec2.internal","openshift-cluster-node-tuning-operator/tuned-dc84m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal","openshift-multus/multus-njm9k","openshift-multus/network-metrics-daemon-5sgzv","openshift-network-diagnostics/network-check-target-7jd4g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4","openshift-dns/node-resolver-l4txq","openshift-image-registry/node-ca-p87w8","openshift-multus/multus-additional-cni-plugins-v9nwx"] Jul 29 20:46:25.438680 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.438662 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:25.438768 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:25.438733 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jd4g" podUID="4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5" Jul 29 20:46:25.440266 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.439809 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.441116 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.441094 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:25.441214 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:25.441151 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhgt5" podUID="ba4aa585-620a-4c88-8c46-09b343186c70" Jul 29 20:46:25.441701 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.441681 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7vbpq\"" Jul 29 20:46:25.442062 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.442041 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 20:46:25.442062 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.442058 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 20:46:25.442196 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.442182 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 20:46:25.442593 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.442577 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mc9jj" Jul 29 20:46:25.443961 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.443902 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.444588 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.444566 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 20:46:25.444696 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.444681 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 20:46:25.444808 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.444792 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-s4jsw\"" Jul 29 20:46:25.445650 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.445458 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.445650 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.445546 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 20:46:25.446039 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.446025 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 20:46:25.446125 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.446060 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 20:46:25.446301 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.446278 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 20:46:25.446405 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.446308 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mcwvv\"" Jul 29 20:46:25.446567 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.446540 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 20:46:25.446658 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.446633 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:25.446729 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:25.446707 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sgzv" podUID="347e06fd-9f35-47dd-9442-ddd628e5bd35" Jul 29 20:46:25.447285 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.447218 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 20:46:25.447778 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.447644 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 20:46:25.447878 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.447784 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 20:46:25.448241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.448042 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fprvm\"" Jul 29 20:46:25.448331 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.448307 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p87w8" Jul 29 20:46:25.451320 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.449767 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.451320 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.449978 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 20:46:25.451320 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.450194 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 20:46:25.451320 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.450304 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 20:46:25.451320 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.450355 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vjl7x\"" Jul 29 20:46:25.452043 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.451989 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 20:46:25.452320 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.452303 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.452428 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.452416 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 20:46:25.452494 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.452430 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 20:46:25.453407 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.453384 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l4txq" Jul 29 20:46:25.454175 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.454151 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 20:46:25.455590 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.455569 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 20:46:25.455877 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.455858 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5wkml\"" Jul 29 20:46:25.456096 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.456081 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 20:46:25.457091 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.456876 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 20:46:25.457221 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.457204 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 20:46:25.457612 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.457595 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 20:46:25.457990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.457973 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-snkvd\"" Jul 29 20:46:25.459239 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.459215 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-lib-modules\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.459824 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.459541 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msl8q\" (UniqueName: \"kubernetes.io/projected/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-kube-api-access-msl8q\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.459824 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.459587 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11939fc-63a4-4f61-9de5-5d9bef849715-system-cni-dir\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.459824 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.459619 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs\") pod \"network-metrics-daemon-5sgzv\" (UID: \"347e06fd-9f35-47dd-9442-ddd628e5bd35\") " pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:25.459824 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.459649 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e6d19b51-bb38-465c-ae75-cfa9d195bb72-serviceca\") pod \"node-ca-p87w8\" (UID: \"e6d19b51-bb38-465c-ae75-cfa9d195bb72\") " pod="openshift-image-registry/node-ca-p87w8" Jul 29 20:46:25.459824 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.459681 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a11939fc-63a4-4f61-9de5-5d9bef849715-cnibin\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.459824 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.459711 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a11939fc-63a4-4f61-9de5-5d9bef849715-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.459824 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.459738 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-run\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.459824 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.459766 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba4aa585-620a-4c88-8c46-09b343186c70-dbus\") pod \"global-pull-secret-syncer-fhgt5\" (UID: \"ba4aa585-620a-4c88-8c46-09b343186c70\") " pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:25.459824 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.459798 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a11939fc-63a4-4f61-9de5-5d9bef849715-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.460351 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.459978 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d172a83f-d86e-4d6b-85ae-792d777676c5-konnectivity-ca\") pod \"konnectivity-agent-mc9jj\" (UID: \"d172a83f-d86e-4d6b-85ae-792d777676c5\") " pod="kube-system/konnectivity-agent-mc9jj" Jul 29 20:46:25.460351 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460044 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-tmp\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.460351 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460078 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04580c8c-3311-478b-b918-debc86d16f14-device-dir\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.460351 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460111 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a11939fc-63a4-4f61-9de5-5d9bef849715-cni-binary-copy\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.460351 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460173 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-kubernetes\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.460351 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460205 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-var-lib-kubelet\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.460351 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460236 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-tuned\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.460351 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460267 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04580c8c-3311-478b-b918-debc86d16f14-socket-dir\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.460351 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460292 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04580c8c-3311-478b-b918-debc86d16f14-etc-selinux\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.460351 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460323 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qvs6\" (UniqueName: \"kubernetes.io/projected/04580c8c-3311-478b-b918-debc86d16f14-kube-api-access-2qvs6\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.460351 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460353 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-host\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.460982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460387 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh6dq\" (UniqueName: \"kubernetes.io/projected/a11939fc-63a4-4f61-9de5-5d9bef849715-kube-api-access-gh6dq\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.460982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460463 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m98sf\" (UniqueName: \"kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf\") pod \"network-check-target-7jd4g\" (UID: \"4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5\") " pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:25.460982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460497 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a11939fc-63a4-4f61-9de5-5d9bef849715-os-release\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.460982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460529 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-modprobe-d\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.460982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460558 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-systemd\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.460982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460585 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04580c8c-3311-478b-b918-debc86d16f14-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.460982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460616 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba4aa585-620a-4c88-8c46-09b343186c70-kubelet-config\") pod \"global-pull-secret-syncer-fhgt5\" (UID: \"ba4aa585-620a-4c88-8c46-09b343186c70\") " pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:25.460982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460653 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-sysctl-d\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.460982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460683 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-sysctl-conf\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.460982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460712 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-sys\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.460982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460736 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04580c8c-3311-478b-b918-debc86d16f14-sys-fs\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.460982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460772 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a11939fc-63a4-4f61-9de5-5d9bef849715-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.460982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460802 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-sysconfig\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.460982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460832 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6d19b51-bb38-465c-ae75-cfa9d195bb72-host\") pod \"node-ca-p87w8\" (UID: \"e6d19b51-bb38-465c-ae75-cfa9d195bb72\") " pod="openshift-image-registry/node-ca-p87w8" Jul 29 20:46:25.460982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460863 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d172a83f-d86e-4d6b-85ae-792d777676c5-agent-certs\") pod \"konnectivity-agent-mc9jj\" (UID: \"d172a83f-d86e-4d6b-85ae-792d777676c5\") " pod="kube-system/konnectivity-agent-mc9jj" Jul 29 20:46:25.460982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460892 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2lw5\" (UniqueName: \"kubernetes.io/projected/347e06fd-9f35-47dd-9442-ddd628e5bd35-kube-api-access-x2lw5\") pod \"network-metrics-daemon-5sgzv\" (UID: \"347e06fd-9f35-47dd-9442-ddd628e5bd35\") " pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:25.460982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.460971 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9w8sf\"" Jul 29 20:46:25.461740 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.461021 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njw56\" (UniqueName: \"kubernetes.io/projected/e6d19b51-bb38-465c-ae75-cfa9d195bb72-kube-api-access-njw56\") pod \"node-ca-p87w8\" (UID: \"e6d19b51-bb38-465c-ae75-cfa9d195bb72\") " pod="openshift-image-registry/node-ca-p87w8" Jul 29 20:46:25.461740 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.461055 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04580c8c-3311-478b-b918-debc86d16f14-registration-dir\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.461740 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.461092 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret\") pod \"global-pull-secret-syncer-fhgt5\" (UID: \"ba4aa585-620a-4c88-8c46-09b343186c70\") " pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:25.462343 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.462151 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5tkhm" Jul 29 20:46:25.463531 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.463508 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 20:46:25.464620 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.464206 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 20:46:25.464620 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.464300 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 20:46:25.464620 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.464376 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hp4td\"" Jul 29 20:46:25.464620 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.464491 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 20:46:25.482908 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.482884 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zv84r" Jul 29 20:46:25.489110 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.489091 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zv84r" Jul 29 20:46:25.548654 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.548626 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 20:46:25.555898 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.555847 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-216.ec2.internal" event={"ID":"4119ef05f02339ab6df3a952dfb37eab","Type":"ContainerStarted","Data":"052e60c7c445330f3940dbff5281c1fa2077974562bf941e5459e88c4aa47141"} Jul 29 20:46:25.558290 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.558247 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal" event={"ID":"4cc20b7e84d827f5d2291ec9a66f9a73","Type":"ContainerStarted","Data":"ffb363ee62b045fb1a4ebf40a3ef583dbfbae5cd52be3c02af5dab8ac7169a10"} Jul 29 20:46:25.561503 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561477 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-host-var-lib-cni-multus\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.561617 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561518 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0f0a3ba3-928b-431b-a554-78a8ecb072f6-iptables-alerter-script\") pod \"iptables-alerter-5tkhm\" (UID: \"0f0a3ba3-928b-431b-a554-78a8ecb072f6\") " pod="openshift-network-operator/iptables-alerter-5tkhm" Jul 29 20:46:25.561617 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561543 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-run-netns\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.561617 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561574 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a11939fc-63a4-4f61-9de5-5d9bef849715-cni-binary-copy\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.561617 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561608 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-kubernetes\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.561809 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561632 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-tuned\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.561809 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561657 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04580c8c-3311-478b-b918-debc86d16f14-etc-selinux\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.561809 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561681 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qvs6\" (UniqueName: \"kubernetes.io/projected/04580c8c-3311-478b-b918-debc86d16f14-kube-api-access-2qvs6\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.561809 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561705 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-cnibin\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.561809 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561727 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f0a3ba3-928b-431b-a554-78a8ecb072f6-host-slash\") pod \"iptables-alerter-5tkhm\" (UID: \"0f0a3ba3-928b-431b-a554-78a8ecb072f6\") " pod="openshift-network-operator/iptables-alerter-5tkhm" Jul 29 20:46:25.561809 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561751 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-var-lib-openvswitch\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.561809 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561781 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-host\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.561809 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561805 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m98sf\" (UniqueName: \"kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf\") pod \"network-check-target-7jd4g\" (UID: \"4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5\") " pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:25.562161 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561830 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a11939fc-63a4-4f61-9de5-5d9bef849715-os-release\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.562161 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561854 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-modprobe-d\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.562161 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561878 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-systemd\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.562161 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561901 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04580c8c-3311-478b-b918-debc86d16f14-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.562161 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561951 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba4aa585-620a-4c88-8c46-09b343186c70-kubelet-config\") pod \"global-pull-secret-syncer-fhgt5\" (UID: \"ba4aa585-620a-4c88-8c46-09b343186c70\") " pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:25.562161 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561975 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-multus-socket-dir-parent\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.562161 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.561999 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-sysctl-conf\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.562161 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562020 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-sys\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.562161 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562042 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-node-log\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.562161 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562071 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-cni-netd\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.562161 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562096 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.562161 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562120 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-sysconfig\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.562161 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562144 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6d19b51-bb38-465c-ae75-cfa9d195bb72-host\") pod \"node-ca-p87w8\" (UID: \"e6d19b51-bb38-465c-ae75-cfa9d195bb72\") " pod="openshift-image-registry/node-ca-p87w8" Jul 29 20:46:25.562745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562168 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-host-var-lib-kubelet\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.562745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562198 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d172a83f-d86e-4d6b-85ae-792d777676c5-agent-certs\") pod \"konnectivity-agent-mc9jj\" (UID: \"d172a83f-d86e-4d6b-85ae-792d777676c5\") " pod="kube-system/konnectivity-agent-mc9jj" Jul 29 20:46:25.562745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562230 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njw56\" (UniqueName: \"kubernetes.io/projected/e6d19b51-bb38-465c-ae75-cfa9d195bb72-kube-api-access-njw56\") pod \"node-ca-p87w8\" (UID: \"e6d19b51-bb38-465c-ae75-cfa9d195bb72\") " pod="openshift-image-registry/node-ca-p87w8" Jul 29 20:46:25.562745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562254 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04580c8c-3311-478b-b918-debc86d16f14-registration-dir\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.562745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562278 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-run-openvswitch\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.562745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562302 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msl8q\" (UniqueName: \"kubernetes.io/projected/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-kube-api-access-msl8q\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.562745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562328 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11939fc-63a4-4f61-9de5-5d9bef849715-system-cni-dir\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.562745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562353 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-multus-conf-dir\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.562745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562382 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgrsn\" (UniqueName: \"kubernetes.io/projected/946ea891-4ad4-4279-843b-c93a2aee2164-kube-api-access-pgrsn\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.562745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562420 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba4aa585-620a-4c88-8c46-09b343186c70-dbus\") pod \"global-pull-secret-syncer-fhgt5\" (UID: \"ba4aa585-620a-4c88-8c46-09b343186c70\") " pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:25.562745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562443 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-system-cni-dir\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.562745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562466 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/43dd2f3d-02e2-47ba-b20a-4365c2c6f45b-hosts-file\") pod \"node-resolver-l4txq\" (UID: \"43dd2f3d-02e2-47ba-b20a-4365c2c6f45b\") " pod="openshift-dns/node-resolver-l4txq" Jul 29 20:46:25.562745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562489 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcqjg\" (UniqueName: \"kubernetes.io/projected/0f0a3ba3-928b-431b-a554-78a8ecb072f6-kube-api-access-mcqjg\") pod \"iptables-alerter-5tkhm\" (UID: \"0f0a3ba3-928b-431b-a554-78a8ecb072f6\") " pod="openshift-network-operator/iptables-alerter-5tkhm" Jul 29 20:46:25.562745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562514 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-etc-openvswitch\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.562745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562537 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/946ea891-4ad4-4279-843b-c93a2aee2164-env-overrides\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.562745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562564 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d172a83f-d86e-4d6b-85ae-792d777676c5-konnectivity-ca\") pod \"konnectivity-agent-mc9jj\" (UID: \"d172a83f-d86e-4d6b-85ae-792d777676c5\") " pod="kube-system/konnectivity-agent-mc9jj" Jul 29 20:46:25.562745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562587 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-tmp\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.563536 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562610 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-run-ovn\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.563536 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562635 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-var-lib-kubelet\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.563536 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562658 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04580c8c-3311-478b-b918-debc86d16f14-socket-dir\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.563536 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562693 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-host-var-lib-cni-bin\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.563536 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562718 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43dd2f3d-02e2-47ba-b20a-4365c2c6f45b-tmp-dir\") pod \"node-resolver-l4txq\" (UID: \"43dd2f3d-02e2-47ba-b20a-4365c2c6f45b\") " pod="openshift-dns/node-resolver-l4txq" Jul 29 20:46:25.563536 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562742 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxhld\" (UniqueName: \"kubernetes.io/projected/43dd2f3d-02e2-47ba-b20a-4365c2c6f45b-kube-api-access-fxhld\") pod \"node-resolver-l4txq\" (UID: \"43dd2f3d-02e2-47ba-b20a-4365c2c6f45b\") " pod="openshift-dns/node-resolver-l4txq" Jul 29 20:46:25.563536 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562766 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-kubelet\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.563536 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562789 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-slash\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.563536 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562813 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/946ea891-4ad4-4279-843b-c93a2aee2164-ovn-node-metrics-cert\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.563536 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562846 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh6dq\" (UniqueName: \"kubernetes.io/projected/a11939fc-63a4-4f61-9de5-5d9bef849715-kube-api-access-gh6dq\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.563536 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562872 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-hostroot\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.563536 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562895 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b1661ddb-eba2-4408-a48b-f2edcc9b600e-multus-daemon-config\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.563536 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562896 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6d19b51-bb38-465c-ae75-cfa9d195bb72-host\") pod \"node-ca-p87w8\" (UID: \"e6d19b51-bb38-465c-ae75-cfa9d195bb72\") " pod="openshift-image-registry/node-ca-p87w8" Jul 29 20:46:25.563536 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562933 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-systemd-units\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.563536 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562960 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-run-ovn-kubernetes\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.563536 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.562987 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-sysctl-d\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.563536 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563025 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04580c8c-3311-478b-b918-debc86d16f14-sys-fs\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.564340 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563050 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/946ea891-4ad4-4279-843b-c93a2aee2164-ovnkube-script-lib\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.564340 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563079 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a11939fc-63a4-4f61-9de5-5d9bef849715-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.564340 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563104 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-os-release\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.564340 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563127 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-etc-kubernetes\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.564340 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563151 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-log-socket\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.564340 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563175 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/946ea891-4ad4-4279-843b-c93a2aee2164-ovnkube-config\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.564340 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563201 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2lw5\" (UniqueName: \"kubernetes.io/projected/347e06fd-9f35-47dd-9442-ddd628e5bd35-kube-api-access-x2lw5\") pod \"network-metrics-daemon-5sgzv\" (UID: \"347e06fd-9f35-47dd-9442-ddd628e5bd35\") " pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:25.564340 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563229 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret\") pod \"global-pull-secret-syncer-fhgt5\" (UID: \"ba4aa585-620a-4c88-8c46-09b343186c70\") " pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:25.564340 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563267 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-host-run-netns\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.564340 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563292 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-host-run-multus-certs\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.564340 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563316 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-cni-bin\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.564340 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563341 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-lib-modules\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.564340 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563374 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs\") pod \"network-metrics-daemon-5sgzv\" (UID: \"347e06fd-9f35-47dd-9442-ddd628e5bd35\") " pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:25.564340 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563399 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e6d19b51-bb38-465c-ae75-cfa9d195bb72-serviceca\") pod \"node-ca-p87w8\" (UID: \"e6d19b51-bb38-465c-ae75-cfa9d195bb72\") " pod="openshift-image-registry/node-ca-p87w8" Jul 29 20:46:25.564340 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563429 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a11939fc-63a4-4f61-9de5-5d9bef849715-cnibin\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.564340 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563456 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a11939fc-63a4-4f61-9de5-5d9bef849715-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.564340 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-run\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.565151 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563504 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj66c\" (UniqueName: \"kubernetes.io/projected/b1661ddb-eba2-4408-a48b-f2edcc9b600e-kube-api-access-kj66c\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.565151 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563529 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-run-systemd\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.565151 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563536 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a11939fc-63a4-4f61-9de5-5d9bef849715-cni-binary-copy\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.565151 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563556 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a11939fc-63a4-4f61-9de5-5d9bef849715-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.565151 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563606 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04580c8c-3311-478b-b918-debc86d16f14-device-dir\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.565151 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563642 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-multus-cni-dir\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.565151 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563669 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1661ddb-eba2-4408-a48b-f2edcc9b600e-cni-binary-copy\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.565151 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563697 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-host-run-k8s-cni-cncf-io\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.565151 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.563790 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-kubernetes\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.565151 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.564037 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a11939fc-63a4-4f61-9de5-5d9bef849715-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.565151 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.564095 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 20:46:25.565975 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.565898 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04580c8c-3311-478b-b918-debc86d16f14-registration-dir\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.566460 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.566173 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11939fc-63a4-4f61-9de5-5d9bef849715-system-cni-dir\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.566460 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.566366 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba4aa585-620a-4c88-8c46-09b343186c70-dbus\") pod \"global-pull-secret-syncer-fhgt5\" (UID: \"ba4aa585-620a-4c88-8c46-09b343186c70\") " pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:25.566678 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.566655 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-sysctl-d\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.566741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.566708 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-var-lib-kubelet\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.566741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.566728 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04580c8c-3311-478b-b918-debc86d16f14-sys-fs\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.567000 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.566976 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d172a83f-d86e-4d6b-85ae-792d777676c5-konnectivity-ca\") pod \"konnectivity-agent-mc9jj\" (UID: \"d172a83f-d86e-4d6b-85ae-792d777676c5\") " pod="kube-system/konnectivity-agent-mc9jj" Jul 29 20:46:25.567432 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.567410 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a11939fc-63a4-4f61-9de5-5d9bef849715-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.567511 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.567457 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a11939fc-63a4-4f61-9de5-5d9bef849715-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.567511 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.567476 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-sys\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.567624 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.567531 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04580c8c-3311-478b-b918-debc86d16f14-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.567682 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.567634 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04580c8c-3311-478b-b918-debc86d16f14-etc-selinux\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.567740 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.567697 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-systemd\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.567740 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.567701 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-modprobe-d\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.567846 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.567772 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba4aa585-620a-4c88-8c46-09b343186c70-kubelet-config\") pod \"global-pull-secret-syncer-fhgt5\" (UID: \"ba4aa585-620a-4c88-8c46-09b343186c70\") " pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:25.567961 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.567943 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-host\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.568016 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.567960 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04580c8c-3311-478b-b918-debc86d16f14-socket-dir\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.568078 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.568039 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a11939fc-63a4-4f61-9de5-5d9bef849715-os-release\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.568308 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:25.568290 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:46:25.568389 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:25.568377 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs podName:347e06fd-9f35-47dd-9442-ddd628e5bd35 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:26.068349447 +0000 UTC m=+2.939452894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs") pod "network-metrics-daemon-5sgzv" (UID: "347e06fd-9f35-47dd-9442-ddd628e5bd35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:46:25.568539 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:25.568525 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 20:46:25.568611 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:25.568589 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret podName:ba4aa585-620a-4c88-8c46-09b343186c70 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:26.068561498 +0000 UTC m=+2.939664945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret") pod "global-pull-secret-syncer-fhgt5" (UID: "ba4aa585-620a-4c88-8c46-09b343186c70") : object "kube-system"/"original-pull-secret" not registered Jul 29 20:46:25.568672 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.568640 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a11939fc-63a4-4f61-9de5-5d9bef849715-cnibin\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.568753 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.568732 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-lib-modules\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.568753 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.568726 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e6d19b51-bb38-465c-ae75-cfa9d195bb72-serviceca\") pod \"node-ca-p87w8\" (UID: \"e6d19b51-bb38-465c-ae75-cfa9d195bb72\") " pod="openshift-image-registry/node-ca-p87w8" Jul 29 20:46:25.568852 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.568791 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-sysconfig\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.568852 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.568797 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04580c8c-3311-478b-b918-debc86d16f14-device-dir\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.568852 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.568846 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-run\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.569161 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.569131 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-sysctl-conf\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.570894 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.570840 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-etc-tuned\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.571764 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.571729 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d172a83f-d86e-4d6b-85ae-792d777676c5-agent-certs\") pod \"konnectivity-agent-mc9jj\" (UID: \"d172a83f-d86e-4d6b-85ae-792d777676c5\") " pod="kube-system/konnectivity-agent-mc9jj" Jul 29 20:46:25.572464 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.572443 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-tmp\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.573693 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.573662 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njw56\" (UniqueName: \"kubernetes.io/projected/e6d19b51-bb38-465c-ae75-cfa9d195bb72-kube-api-access-njw56\") pod \"node-ca-p87w8\" (UID: \"e6d19b51-bb38-465c-ae75-cfa9d195bb72\") " pod="openshift-image-registry/node-ca-p87w8" Jul 29 20:46:25.573908 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:25.573889 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 20:46:25.574631 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:25.574615 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 20:46:25.574715 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:25.574639 2562 projected.go:196] Error preparing data for projected volume kube-api-access-m98sf for pod openshift-network-diagnostics/network-check-target-7jd4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:46:25.574772 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:25.574725 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf podName:4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:26.074707967 +0000 UTC m=+2.945811428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m98sf" (UniqueName: "kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf") pod "network-check-target-7jd4g" (UID: "4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:46:25.574875 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.574855 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msl8q\" (UniqueName: \"kubernetes.io/projected/744f3fb4-81ae-4b9c-9bc3-7fb975c91336-kube-api-access-msl8q\") pod \"tuned-dc84m\" (UID: \"744f3fb4-81ae-4b9c-9bc3-7fb975c91336\") " pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.576497 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.576458 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2lw5\" (UniqueName: \"kubernetes.io/projected/347e06fd-9f35-47dd-9442-ddd628e5bd35-kube-api-access-x2lw5\") pod \"network-metrics-daemon-5sgzv\" (UID: \"347e06fd-9f35-47dd-9442-ddd628e5bd35\") " pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:25.576761 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.576735 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qvs6\" (UniqueName: \"kubernetes.io/projected/04580c8c-3311-478b-b918-debc86d16f14-kube-api-access-2qvs6\") pod \"aws-ebs-csi-driver-node-b4db4\" (UID: \"04580c8c-3311-478b-b918-debc86d16f14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.582742 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.579411 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh6dq\" (UniqueName: \"kubernetes.io/projected/a11939fc-63a4-4f61-9de5-5d9bef849715-kube-api-access-gh6dq\") pod \"multus-additional-cni-plugins-v9nwx\" (UID: \"a11939fc-63a4-4f61-9de5-5d9bef849715\") " pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.666741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.664636 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-run-ovn\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.666741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.664686 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-host-var-lib-cni-bin\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.666741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.664713 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43dd2f3d-02e2-47ba-b20a-4365c2c6f45b-tmp-dir\") pod \"node-resolver-l4txq\" (UID: \"43dd2f3d-02e2-47ba-b20a-4365c2c6f45b\") " pod="openshift-dns/node-resolver-l4txq" Jul 29 20:46:25.666741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.664737 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxhld\" (UniqueName: \"kubernetes.io/projected/43dd2f3d-02e2-47ba-b20a-4365c2c6f45b-kube-api-access-fxhld\") pod \"node-resolver-l4txq\" (UID: \"43dd2f3d-02e2-47ba-b20a-4365c2c6f45b\") " pod="openshift-dns/node-resolver-l4txq" Jul 29 20:46:25.666741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.664760 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-kubelet\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.666741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.664782 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-slash\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.666741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.664812 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/946ea891-4ad4-4279-843b-c93a2aee2164-ovn-node-metrics-cert\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.666741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.664836 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-hostroot\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.666741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.664857 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b1661ddb-eba2-4408-a48b-f2edcc9b600e-multus-daemon-config\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.666741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.664879 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-systemd-units\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.666741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.664903 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-run-ovn-kubernetes\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.666741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.664911 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-run-ovn\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.666741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665189 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-host-var-lib-cni-bin\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.666741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.664944 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/946ea891-4ad4-4279-843b-c93a2aee2164-ovnkube-script-lib\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.666741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665217 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43dd2f3d-02e2-47ba-b20a-4365c2c6f45b-tmp-dir\") pod \"node-resolver-l4txq\" (UID: \"43dd2f3d-02e2-47ba-b20a-4365c2c6f45b\") " pod="openshift-dns/node-resolver-l4txq" Jul 29 20:46:25.666741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665248 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-slash\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.666741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665262 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-os-release\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.666741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665294 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-etc-kubernetes\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.667705 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665318 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-log-socket\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.667705 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665341 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/946ea891-4ad4-4279-843b-c93a2aee2164-ovnkube-config\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.667705 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665381 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-host-run-netns\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.667705 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665404 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-host-run-multus-certs\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.667705 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665428 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-cni-bin\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.667705 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665474 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj66c\" (UniqueName: \"kubernetes.io/projected/b1661ddb-eba2-4408-a48b-f2edcc9b600e-kube-api-access-kj66c\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.667705 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665494 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-run-systemd\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.667705 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665515 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-multus-cni-dir\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.667705 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665551 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1661ddb-eba2-4408-a48b-f2edcc9b600e-cni-binary-copy\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.667705 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665579 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-host-run-k8s-cni-cncf-io\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.667705 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665604 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-host-var-lib-cni-multus\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.667705 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665624 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/946ea891-4ad4-4279-843b-c93a2aee2164-ovnkube-script-lib\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.667705 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665627 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0f0a3ba3-928b-431b-a554-78a8ecb072f6-iptables-alerter-script\") pod \"iptables-alerter-5tkhm\" (UID: \"0f0a3ba3-928b-431b-a554-78a8ecb072f6\") " pod="openshift-network-operator/iptables-alerter-5tkhm" Jul 29 20:46:25.667705 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665218 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-kubelet\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.667705 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665673 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-run-netns\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.667705 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665685 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-systemd-units\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.667705 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665706 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-cnibin\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.667705 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665714 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-os-release\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.668607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665731 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f0a3ba3-928b-431b-a554-78a8ecb072f6-host-slash\") pod \"iptables-alerter-5tkhm\" (UID: \"0f0a3ba3-928b-431b-a554-78a8ecb072f6\") " pod="openshift-network-operator/iptables-alerter-5tkhm" Jul 29 20:46:25.668607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665753 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-etc-kubernetes\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.668607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665757 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-var-lib-openvswitch\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.668607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665752 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b1661ddb-eba2-4408-a48b-f2edcc9b600e-multus-daemon-config\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.668607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665792 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-run-ovn-kubernetes\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.668607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665802 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-log-socket\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.668607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665805 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-multus-socket-dir-parent\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.668607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665844 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-host-run-multus-certs\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.668607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665848 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-cnibin\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.668607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665858 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-multus-socket-dir-parent\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.668607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665879 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-node-log\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.668607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665897 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f0a3ba3-928b-431b-a554-78a8ecb072f6-host-slash\") pod \"iptables-alerter-5tkhm\" (UID: \"0f0a3ba3-928b-431b-a554-78a8ecb072f6\") " pod="openshift-network-operator/iptables-alerter-5tkhm" Jul 29 20:46:25.668607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-cni-netd\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.668607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665954 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.668607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665986 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-host-var-lib-kubelet\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.668607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666016 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-run-openvswitch\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.668607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666043 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-multus-conf-dir\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.668607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666068 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgrsn\" (UniqueName: \"kubernetes.io/projected/946ea891-4ad4-4279-843b-c93a2aee2164-kube-api-access-pgrsn\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.669472 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666098 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-system-cni-dir\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.669472 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666121 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/43dd2f3d-02e2-47ba-b20a-4365c2c6f45b-hosts-file\") pod \"node-resolver-l4txq\" (UID: \"43dd2f3d-02e2-47ba-b20a-4365c2c6f45b\") " pod="openshift-dns/node-resolver-l4txq" Jul 29 20:46:25.669472 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666169 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-host-run-k8s-cni-cncf-io\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.669472 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666193 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0f0a3ba3-928b-431b-a554-78a8ecb072f6-iptables-alerter-script\") pod \"iptables-alerter-5tkhm\" (UID: \"0f0a3ba3-928b-431b-a554-78a8ecb072f6\") " pod="openshift-network-operator/iptables-alerter-5tkhm" Jul 29 20:46:25.669472 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666228 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-host-var-lib-cni-multus\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.669472 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666239 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcqjg\" (UniqueName: \"kubernetes.io/projected/0f0a3ba3-928b-431b-a554-78a8ecb072f6-kube-api-access-mcqjg\") pod \"iptables-alerter-5tkhm\" (UID: \"0f0a3ba3-928b-431b-a554-78a8ecb072f6\") " pod="openshift-network-operator/iptables-alerter-5tkhm" Jul 29 20:46:25.669472 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666266 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-etc-openvswitch\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.669472 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666277 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-run-systemd\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.669472 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666291 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/946ea891-4ad4-4279-843b-c93a2aee2164-env-overrides\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.669472 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666337 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-multus-cni-dir\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.669472 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666367 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1661ddb-eba2-4408-a48b-f2edcc9b600e-cni-binary-copy\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.669472 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666375 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-hostroot\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.669472 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666415 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.669472 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666452 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-host-run-netns\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.669472 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666487 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-node-log\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.669472 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.665881 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-cni-bin\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.669472 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666549 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-cni-netd\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.669472 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666587 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-var-lib-openvswitch\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.670326 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666595 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/946ea891-4ad4-4279-843b-c93a2aee2164-ovnkube-config\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.670326 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666601 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-multus-conf-dir\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.670326 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666647 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-host-run-netns\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.670326 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666652 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-run-openvswitch\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.670326 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666665 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/946ea891-4ad4-4279-843b-c93a2aee2164-env-overrides\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.670326 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666180 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/43dd2f3d-02e2-47ba-b20a-4365c2c6f45b-hosts-file\") pod \"node-resolver-l4txq\" (UID: \"43dd2f3d-02e2-47ba-b20a-4365c2c6f45b\") " pod="openshift-dns/node-resolver-l4txq" Jul 29 20:46:25.670326 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666698 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/946ea891-4ad4-4279-843b-c93a2aee2164-etc-openvswitch\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.670326 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666702 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-system-cni-dir\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.670326 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.666745 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1661ddb-eba2-4408-a48b-f2edcc9b600e-host-var-lib-kubelet\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.670326 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.667829 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/946ea891-4ad4-4279-843b-c93a2aee2164-ovn-node-metrics-cert\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.672726 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.672700 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxhld\" (UniqueName: \"kubernetes.io/projected/43dd2f3d-02e2-47ba-b20a-4365c2c6f45b-kube-api-access-fxhld\") pod \"node-resolver-l4txq\" (UID: \"43dd2f3d-02e2-47ba-b20a-4365c2c6f45b\") " pod="openshift-dns/node-resolver-l4txq" Jul 29 20:46:25.673856 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.673832 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj66c\" (UniqueName: \"kubernetes.io/projected/b1661ddb-eba2-4408-a48b-f2edcc9b600e-kube-api-access-kj66c\") pod \"multus-njm9k\" (UID: \"b1661ddb-eba2-4408-a48b-f2edcc9b600e\") " pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.674647 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.674493 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcqjg\" (UniqueName: \"kubernetes.io/projected/0f0a3ba3-928b-431b-a554-78a8ecb072f6-kube-api-access-mcqjg\") pod \"iptables-alerter-5tkhm\" (UID: \"0f0a3ba3-928b-431b-a554-78a8ecb072f6\") " pod="openshift-network-operator/iptables-alerter-5tkhm" Jul 29 20:46:25.674647 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.674587 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgrsn\" (UniqueName: \"kubernetes.io/projected/946ea891-4ad4-4279-843b-c93a2aee2164-kube-api-access-pgrsn\") pod \"ovnkube-node-tplf2\" (UID: \"946ea891-4ad4-4279-843b-c93a2aee2164\") " pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.760721 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.760631 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" Jul 29 20:46:25.764585 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.764561 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mc9jj" Jul 29 20:46:25.774906 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.774888 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v9nwx" Jul 29 20:46:25.781465 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.781437 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dc84m" Jul 29 20:46:25.789000 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.788979 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p87w8" Jul 29 20:46:25.795723 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.795698 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:25.803337 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.803316 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-njm9k" Jul 29 20:46:25.816953 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.816914 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l4txq" Jul 29 20:46:25.824587 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.824563 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5tkhm" Jul 29 20:46:25.981701 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:25.981670 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 20:46:26.068745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:26.068647 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret\") pod \"global-pull-secret-syncer-fhgt5\" (UID: \"ba4aa585-620a-4c88-8c46-09b343186c70\") " pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:26.068745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:26.068692 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs\") pod \"network-metrics-daemon-5sgzv\" (UID: \"347e06fd-9f35-47dd-9442-ddd628e5bd35\") " pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:26.069010 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:26.068826 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 20:46:26.069010 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:26.068884 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret podName:ba4aa585-620a-4c88-8c46-09b343186c70 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:27.068870111 +0000 UTC m=+3.939973559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret") pod "global-pull-secret-syncer-fhgt5" (UID: "ba4aa585-620a-4c88-8c46-09b343186c70") : object "kube-system"/"original-pull-secret" not registered Jul 29 20:46:26.069010 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:26.068897 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:46:26.069010 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:26.068976 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs podName:347e06fd-9f35-47dd-9442-ddd628e5bd35 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:27.068958215 +0000 UTC m=+3.940061674 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs") pod "network-metrics-daemon-5sgzv" (UID: "347e06fd-9f35-47dd-9442-ddd628e5bd35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:46:26.169457 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:26.169426 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m98sf\" (UniqueName: \"kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf\") pod \"network-check-target-7jd4g\" (UID: \"4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5\") " pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:26.169627 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:26.169607 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 20:46:26.169669 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:26.169632 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 20:46:26.169669 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:26.169642 2562 projected.go:196] Error preparing data for projected volume kube-api-access-m98sf for pod openshift-network-diagnostics/network-check-target-7jd4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:46:26.169735 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:26.169692 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf podName:4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:27.16967868 +0000 UTC m=+4.040782125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m98sf" (UniqueName: "kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf") pod "network-check-target-7jd4g" (UID: "4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:46:26.492003 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:26.491972 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 20:41:25 +0000 UTC" deadline="2028-04-20 05:01:51.789891736 +0000 UTC" Jul 29 20:46:26.492003 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:26.492000 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15128h15m25.297894886s" Jul 29 20:46:26.507290 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:26.507260 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd172a83f_d86e_4d6b_85ae_792d777676c5.slice/crio-0bc525f39271796b229f964a90628c9d87ace7b396dbd95afd976dee796844a9 WatchSource:0}: Error finding container 0bc525f39271796b229f964a90628c9d87ace7b396dbd95afd976dee796844a9: Status 404 returned error can't find the container with id 0bc525f39271796b229f964a90628c9d87ace7b396dbd95afd976dee796844a9 Jul 29 20:46:26.520615 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:26.520589 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04580c8c_3311_478b_b918_debc86d16f14.slice/crio-fa6094d1d9f95babb77b33d609afe0d071fbcdb717082154a7c9e0c914820b6d WatchSource:0}: Error finding container fa6094d1d9f95babb77b33d609afe0d071fbcdb717082154a7c9e0c914820b6d: Status 404 returned error can't find the container with id fa6094d1d9f95babb77b33d609afe0d071fbcdb717082154a7c9e0c914820b6d Jul 29 20:46:26.521669 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:26.521628 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946ea891_4ad4_4279_843b_c93a2aee2164.slice/crio-61ea8bb91a8745732e772c8cbce9e86b502810cd196b5bc3cbefc0f898313ec5 WatchSource:0}: Error finding container 61ea8bb91a8745732e772c8cbce9e86b502810cd196b5bc3cbefc0f898313ec5: Status 404 returned error can't find the container with id 61ea8bb91a8745732e772c8cbce9e86b502810cd196b5bc3cbefc0f898313ec5 Jul 29 20:46:26.522857 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:26.522834 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11939fc_63a4_4f61_9de5_5d9bef849715.slice/crio-f6b31716c409029cd8c92662e74f0253b95d3c227eecb0ba7bb640c06ece71ca WatchSource:0}: Error finding container f6b31716c409029cd8c92662e74f0253b95d3c227eecb0ba7bb640c06ece71ca: Status 404 returned error can't find the container with id f6b31716c409029cd8c92662e74f0253b95d3c227eecb0ba7bb640c06ece71ca Jul 29 20:46:26.525228 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:26.525197 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d19b51_bb38_465c_ae75_cfa9d195bb72.slice/crio-344ca2fc7e98269d3b2fd3b29b964093edfe39ac1b81f49ad407705fc32faf86 WatchSource:0}: Error finding container 344ca2fc7e98269d3b2fd3b29b964093edfe39ac1b81f49ad407705fc32faf86: Status 404 returned error can't find the container with id 344ca2fc7e98269d3b2fd3b29b964093edfe39ac1b81f49ad407705fc32faf86 Jul 29 20:46:26.526428 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:26.526407 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43dd2f3d_02e2_47ba_b20a_4365c2c6f45b.slice/crio-1704ceb97924c01a24d3fa05ba3cec147debdf8437153580657b320593125554 WatchSource:0}: Error finding container 1704ceb97924c01a24d3fa05ba3cec147debdf8437153580657b320593125554: Status 404 returned error can't find the container with id 1704ceb97924c01a24d3fa05ba3cec147debdf8437153580657b320593125554 Jul 29 20:46:26.527959 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:26.527944 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1661ddb_eba2_4408_a48b_f2edcc9b600e.slice/crio-eabb0167f2d3c167f32f6fed617857a7a2b84537514ea484cd1c8ba42d8874cd WatchSource:0}: Error finding container eabb0167f2d3c167f32f6fed617857a7a2b84537514ea484cd1c8ba42d8874cd: Status 404 returned error can't find the container with id eabb0167f2d3c167f32f6fed617857a7a2b84537514ea484cd1c8ba42d8874cd Jul 29 20:46:26.530383 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:26.529182 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0a3ba3_928b_431b_a554_78a8ecb072f6.slice/crio-c6f292dc16c39ddd754f55fc5fa1c844451cbef06511fea3ab5cc67fe338ebe2 WatchSource:0}: Error finding container c6f292dc16c39ddd754f55fc5fa1c844451cbef06511fea3ab5cc67fe338ebe2: Status 404 returned error can't find the container with id c6f292dc16c39ddd754f55fc5fa1c844451cbef06511fea3ab5cc67fe338ebe2 Jul 29 20:46:26.530383 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:46:26.529752 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744f3fb4_81ae_4b9c_9bc3_7fb975c91336.slice/crio-aaf187bfb99b1fd5a1dc4bfa09acfc7e47d5fd15ada1ecedd3aacf93b8716b55 WatchSource:0}: Error finding container aaf187bfb99b1fd5a1dc4bfa09acfc7e47d5fd15ada1ecedd3aacf93b8716b55: Status 404 returned error can't find the container with id aaf187bfb99b1fd5a1dc4bfa09acfc7e47d5fd15ada1ecedd3aacf93b8716b55 Jul 29 20:46:26.551126 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:26.551102 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:26.551223 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:26.551205 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jd4g" podUID="4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5" Jul 29 20:46:26.561487 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:26.561460 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dc84m" event={"ID":"744f3fb4-81ae-4b9c-9bc3-7fb975c91336","Type":"ContainerStarted","Data":"aaf187bfb99b1fd5a1dc4bfa09acfc7e47d5fd15ada1ecedd3aacf93b8716b55"} Jul 29 20:46:26.562499 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:26.562477 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-njm9k" event={"ID":"b1661ddb-eba2-4408-a48b-f2edcc9b600e","Type":"ContainerStarted","Data":"eabb0167f2d3c167f32f6fed617857a7a2b84537514ea484cd1c8ba42d8874cd"} Jul 29 20:46:26.563523 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:26.563501 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l4txq" event={"ID":"43dd2f3d-02e2-47ba-b20a-4365c2c6f45b","Type":"ContainerStarted","Data":"1704ceb97924c01a24d3fa05ba3cec147debdf8437153580657b320593125554"} Jul 29 20:46:26.567043 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:26.567016 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9nwx" event={"ID":"a11939fc-63a4-4f61-9de5-5d9bef849715","Type":"ContainerStarted","Data":"f6b31716c409029cd8c92662e74f0253b95d3c227eecb0ba7bb640c06ece71ca"} Jul 29 20:46:26.570148 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:26.570079 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" event={"ID":"04580c8c-3311-478b-b918-debc86d16f14","Type":"ContainerStarted","Data":"fa6094d1d9f95babb77b33d609afe0d071fbcdb717082154a7c9e0c914820b6d"} Jul 29 20:46:26.572197 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:26.572162 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5tkhm" event={"ID":"0f0a3ba3-928b-431b-a554-78a8ecb072f6","Type":"ContainerStarted","Data":"c6f292dc16c39ddd754f55fc5fa1c844451cbef06511fea3ab5cc67fe338ebe2"} Jul 29 20:46:26.573315 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:26.573291 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p87w8" event={"ID":"e6d19b51-bb38-465c-ae75-cfa9d195bb72","Type":"ContainerStarted","Data":"344ca2fc7e98269d3b2fd3b29b964093edfe39ac1b81f49ad407705fc32faf86"} Jul 29 20:46:26.574475 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:26.574445 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" event={"ID":"946ea891-4ad4-4279-843b-c93a2aee2164","Type":"ContainerStarted","Data":"61ea8bb91a8745732e772c8cbce9e86b502810cd196b5bc3cbefc0f898313ec5"} Jul 29 20:46:26.575549 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:26.575528 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mc9jj" event={"ID":"d172a83f-d86e-4d6b-85ae-792d777676c5","Type":"ContainerStarted","Data":"0bc525f39271796b229f964a90628c9d87ace7b396dbd95afd976dee796844a9"} Jul 29 20:46:27.075359 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:27.075037 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret\") pod \"global-pull-secret-syncer-fhgt5\" (UID: \"ba4aa585-620a-4c88-8c46-09b343186c70\") " pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:27.075554 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:27.075395 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs\") pod \"network-metrics-daemon-5sgzv\" (UID: \"347e06fd-9f35-47dd-9442-ddd628e5bd35\") " pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:27.075554 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:27.075527 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:46:27.075660 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:27.075593 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs podName:347e06fd-9f35-47dd-9442-ddd628e5bd35 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:29.075575116 +0000 UTC m=+5.946678562 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs") pod "network-metrics-daemon-5sgzv" (UID: "347e06fd-9f35-47dd-9442-ddd628e5bd35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:46:27.076028 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:27.076005 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 20:46:27.076128 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:27.076065 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret podName:ba4aa585-620a-4c88-8c46-09b343186c70 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:29.076048477 +0000 UTC m=+5.947151933 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret") pod "global-pull-secret-syncer-fhgt5" (UID: "ba4aa585-620a-4c88-8c46-09b343186c70") : object "kube-system"/"original-pull-secret" not registered Jul 29 20:46:27.177103 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:27.176430 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m98sf\" (UniqueName: \"kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf\") pod \"network-check-target-7jd4g\" (UID: \"4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5\") " pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:27.177103 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:27.176619 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 20:46:27.177103 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:27.176643 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 20:46:27.177103 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:27.176656 2562 projected.go:196] Error preparing data for projected volume kube-api-access-m98sf for pod openshift-network-diagnostics/network-check-target-7jd4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:46:27.177103 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:27.176717 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf podName:4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:29.176699106 +0000 UTC m=+6.047802567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m98sf" (UniqueName: "kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf") pod "network-check-target-7jd4g" (UID: "4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:46:27.493237 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:27.493073 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 20:41:25 +0000 UTC" deadline="2028-03-30 12:47:09.333772785 +0000 UTC" Jul 29 20:46:27.493237 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:27.493132 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14632h0m41.840644362s" Jul 29 20:46:27.551312 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:27.550829 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:27.551312 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:27.550983 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sgzv" podUID="347e06fd-9f35-47dd-9442-ddd628e5bd35" Jul 29 20:46:27.551539 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:27.551352 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:27.551539 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:27.551453 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhgt5" podUID="ba4aa585-620a-4c88-8c46-09b343186c70" Jul 29 20:46:27.580218 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:27.580175 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-216.ec2.internal" event={"ID":"4119ef05f02339ab6df3a952dfb37eab","Type":"ContainerStarted","Data":"1ecbd4093e1ba5bfc20ebfcb7004408396e642de2fbd06ed38db12d3eca15324"} Jul 29 20:46:27.591055 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:27.591016 2562 generic.go:356] "Generic (PLEG): container finished" podID="4cc20b7e84d827f5d2291ec9a66f9a73" containerID="108daeaf70427fd280d9a91d801148843edde7c9370affcbb09bc7613370810d" exitCode=0 Jul 29 20:46:27.591055 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:27.591058 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal" event={"ID":"4cc20b7e84d827f5d2291ec9a66f9a73","Type":"ContainerDied","Data":"108daeaf70427fd280d9a91d801148843edde7c9370affcbb09bc7613370810d"} Jul 29 20:46:27.608310 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:27.608259 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-216.ec2.internal" podStartSLOduration=3.608239523 podStartE2EDuration="3.608239523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:27.594295998 +0000 UTC m=+4.465399468" watchObservedRunningTime="2026-07-29 20:46:27.608239523 +0000 UTC m=+4.479342990" Jul 29 20:46:28.551441 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:28.551402 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:28.551969 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:28.551543 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jd4g" podUID="4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5" Jul 29 20:46:28.615625 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:28.615576 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal" event={"ID":"4cc20b7e84d827f5d2291ec9a66f9a73","Type":"ContainerStarted","Data":"f96d5bd8a586f9a747e1f8c2fdfd716d11741d3e484a461b29b52ec39f23b333"} Jul 29 20:46:29.095382 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:29.095344 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret\") pod \"global-pull-secret-syncer-fhgt5\" (UID: \"ba4aa585-620a-4c88-8c46-09b343186c70\") " pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:29.095570 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:29.095398 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs\") pod \"network-metrics-daemon-5sgzv\" (UID: \"347e06fd-9f35-47dd-9442-ddd628e5bd35\") " pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:29.095570 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:29.095541 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:46:29.095684 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:29.095602 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs podName:347e06fd-9f35-47dd-9442-ddd628e5bd35 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:33.095583715 +0000 UTC m=+9.966687166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs") pod "network-metrics-daemon-5sgzv" (UID: "347e06fd-9f35-47dd-9442-ddd628e5bd35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:46:29.096000 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:29.095844 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 20:46:29.096000 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:29.095906 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret podName:ba4aa585-620a-4c88-8c46-09b343186c70 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:33.095886743 +0000 UTC m=+9.966990198 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret") pod "global-pull-secret-syncer-fhgt5" (UID: "ba4aa585-620a-4c88-8c46-09b343186c70") : object "kube-system"/"original-pull-secret" not registered Jul 29 20:46:29.196873 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:29.196395 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m98sf\" (UniqueName: \"kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf\") pod \"network-check-target-7jd4g\" (UID: \"4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5\") " pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:29.196873 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:29.196553 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 20:46:29.196873 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:29.196571 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 20:46:29.196873 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:29.196580 2562 projected.go:196] Error preparing data for projected volume kube-api-access-m98sf for pod openshift-network-diagnostics/network-check-target-7jd4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:46:29.196873 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:29.196623 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf podName:4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:33.196610172 +0000 UTC m=+10.067713618 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m98sf" (UniqueName: "kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf") pod "network-check-target-7jd4g" (UID: "4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:46:29.551645 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:29.551564 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:29.552114 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:29.551814 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sgzv" podUID="347e06fd-9f35-47dd-9442-ddd628e5bd35" Jul 29 20:46:29.552266 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:29.552244 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:29.552375 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:29.552345 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhgt5" podUID="ba4aa585-620a-4c88-8c46-09b343186c70" Jul 29 20:46:30.551684 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:30.551620 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:30.552187 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:30.551817 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jd4g" podUID="4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5" Jul 29 20:46:31.553478 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:31.552431 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:31.553478 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:31.552470 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:31.553478 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:31.552573 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhgt5" podUID="ba4aa585-620a-4c88-8c46-09b343186c70" Jul 29 20:46:31.553478 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:31.552659 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sgzv" podUID="347e06fd-9f35-47dd-9442-ddd628e5bd35" Jul 29 20:46:32.551246 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:32.551207 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:32.551428 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:32.551353 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jd4g" podUID="4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5" Jul 29 20:46:33.128330 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:33.128290 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret\") pod \"global-pull-secret-syncer-fhgt5\" (UID: \"ba4aa585-620a-4c88-8c46-09b343186c70\") " pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:33.128836 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:33.128348 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs\") pod \"network-metrics-daemon-5sgzv\" (UID: \"347e06fd-9f35-47dd-9442-ddd628e5bd35\") " pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:33.128836 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:33.128511 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:46:33.128836 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:33.128575 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs podName:347e06fd-9f35-47dd-9442-ddd628e5bd35 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:41.128555957 +0000 UTC m=+17.999659406 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs") pod "network-metrics-daemon-5sgzv" (UID: "347e06fd-9f35-47dd-9442-ddd628e5bd35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:46:33.129038 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:33.128988 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 20:46:33.129038 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:33.129038 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret podName:ba4aa585-620a-4c88-8c46-09b343186c70 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:41.129022756 +0000 UTC m=+18.000126206 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret") pod "global-pull-secret-syncer-fhgt5" (UID: "ba4aa585-620a-4c88-8c46-09b343186c70") : object "kube-system"/"original-pull-secret" not registered Jul 29 20:46:33.229956 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:33.229707 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m98sf\" (UniqueName: \"kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf\") pod \"network-check-target-7jd4g\" (UID: \"4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5\") " pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:33.229956 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:33.229874 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 20:46:33.229956 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:33.229901 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 20:46:33.229956 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:33.229914 2562 projected.go:196] Error preparing data for projected volume kube-api-access-m98sf for pod openshift-network-diagnostics/network-check-target-7jd4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:46:33.230320 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:33.229999 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf podName:4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:41.229979043 +0000 UTC m=+18.101082494 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m98sf" (UniqueName: "kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf") pod "network-check-target-7jd4g" (UID: "4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:46:33.552602 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:33.552525 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:33.552726 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:33.552651 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sgzv" podUID="347e06fd-9f35-47dd-9442-ddd628e5bd35" Jul 29 20:46:33.553086 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:33.552933 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:33.553086 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:33.553046 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhgt5" podUID="ba4aa585-620a-4c88-8c46-09b343186c70" Jul 29 20:46:34.551718 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:34.551685 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:34.552239 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:34.551824 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jd4g" podUID="4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5" Jul 29 20:46:35.551663 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:35.551617 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:35.551959 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:35.551666 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:35.551959 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:35.551760 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sgzv" podUID="347e06fd-9f35-47dd-9442-ddd628e5bd35" Jul 29 20:46:35.551959 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:35.551914 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhgt5" podUID="ba4aa585-620a-4c88-8c46-09b343186c70" Jul 29 20:46:36.551163 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:36.551128 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:36.551348 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:36.551242 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jd4g" podUID="4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5" Jul 29 20:46:37.554431 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:37.554360 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:37.554824 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:37.554491 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhgt5" podUID="ba4aa585-620a-4c88-8c46-09b343186c70" Jul 29 20:46:37.554824 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:37.554546 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:37.554824 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:37.554640 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sgzv" podUID="347e06fd-9f35-47dd-9442-ddd628e5bd35" Jul 29 20:46:38.551569 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:38.551530 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:38.551792 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:38.551671 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jd4g" podUID="4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5" Jul 29 20:46:39.551203 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:39.551163 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:39.551672 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:39.551214 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:39.551672 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:39.551318 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sgzv" podUID="347e06fd-9f35-47dd-9442-ddd628e5bd35" Jul 29 20:46:39.551672 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:39.551458 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhgt5" podUID="ba4aa585-620a-4c88-8c46-09b343186c70" Jul 29 20:46:40.551003 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:40.550971 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:40.551204 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:40.551086 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jd4g" podUID="4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5" Jul 29 20:46:41.189520 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:41.189490 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret\") pod \"global-pull-secret-syncer-fhgt5\" (UID: \"ba4aa585-620a-4c88-8c46-09b343186c70\") " pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:41.189520 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:41.189525 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs\") pod \"network-metrics-daemon-5sgzv\" (UID: \"347e06fd-9f35-47dd-9442-ddd628e5bd35\") " pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:41.190042 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:41.189633 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:46:41.190042 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:41.189690 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs podName:347e06fd-9f35-47dd-9442-ddd628e5bd35 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:57.189674005 +0000 UTC m=+34.060777451 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs") pod "network-metrics-daemon-5sgzv" (UID: "347e06fd-9f35-47dd-9442-ddd628e5bd35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:46:41.190042 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:41.189633 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 20:46:41.190042 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:41.189757 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret podName:ba4aa585-620a-4c88-8c46-09b343186c70 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:57.189744193 +0000 UTC m=+34.060847655 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret") pod "global-pull-secret-syncer-fhgt5" (UID: "ba4aa585-620a-4c88-8c46-09b343186c70") : object "kube-system"/"original-pull-secret" not registered Jul 29 20:46:41.289980 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:41.289946 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m98sf\" (UniqueName: \"kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf\") pod \"network-check-target-7jd4g\" (UID: \"4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5\") " pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:41.290149 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:41.290077 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 20:46:41.290149 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:41.290090 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 20:46:41.290149 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:41.290101 2562 projected.go:196] Error preparing data for projected volume kube-api-access-m98sf for pod openshift-network-diagnostics/network-check-target-7jd4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:46:41.290266 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:41.290161 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf podName:4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:57.290146674 +0000 UTC m=+34.161250119 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m98sf" (UniqueName: "kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf") pod "network-check-target-7jd4g" (UID: "4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:46:41.551643 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:41.551600 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:41.551789 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:41.551743 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sgzv" podUID="347e06fd-9f35-47dd-9442-ddd628e5bd35" Jul 29 20:46:41.551861 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:41.551791 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:41.551914 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:41.551889 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhgt5" podUID="ba4aa585-620a-4c88-8c46-09b343186c70" Jul 29 20:46:42.551441 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:42.551406 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:42.551939 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:42.551517 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jd4g" podUID="4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5" Jul 29 20:46:43.551712 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:43.551673 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:43.552129 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:43.551771 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhgt5" podUID="ba4aa585-620a-4c88-8c46-09b343186c70" Jul 29 20:46:43.552129 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:43.551876 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:43.552129 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:43.552015 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sgzv" podUID="347e06fd-9f35-47dd-9442-ddd628e5bd35" Jul 29 20:46:44.550705 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:44.550675 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:44.550905 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:44.550794 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jd4g" podUID="4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5" Jul 29 20:46:45.551452 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:45.551425 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:45.551883 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:45.551429 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:45.551883 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:45.551570 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sgzv" podUID="347e06fd-9f35-47dd-9442-ddd628e5bd35" Jul 29 20:46:45.551883 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:45.551609 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhgt5" podUID="ba4aa585-620a-4c88-8c46-09b343186c70" Jul 29 20:46:45.652261 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:45.652050 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l4txq" event={"ID":"43dd2f3d-02e2-47ba-b20a-4365c2c6f45b","Type":"ContainerStarted","Data":"51f50e36f724070c795d5c7e3cd43ce9657556fba387b2da9ea625b195ba8b61"} Jul 29 20:46:45.654354 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:45.654323 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" event={"ID":"04580c8c-3311-478b-b918-debc86d16f14","Type":"ContainerStarted","Data":"f65e2360d56b6cbb2046fedd89122160b877b9bd9510091e516e36d29925bc3b"} Jul 29 20:46:45.655981 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:45.655954 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p87w8" event={"ID":"e6d19b51-bb38-465c-ae75-cfa9d195bb72","Type":"ContainerStarted","Data":"7da8f909eb3018d2ebabcc5cb1f7969d4ece900c89ad3fcafce9b074753c20e0"} Jul 29 20:46:45.657442 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:45.657223 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mc9jj" event={"ID":"d172a83f-d86e-4d6b-85ae-792d777676c5","Type":"ContainerStarted","Data":"758a23e4eff895c2513917b7c2cc7e04a6059b97b2434d7defdaf1adadb10bea"} Jul 29 20:46:45.666127 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:45.666080 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-216.ec2.internal" podStartSLOduration=21.666061945 podStartE2EDuration="21.666061945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:28.629561151 +0000 UTC m=+5.500664620" watchObservedRunningTime="2026-07-29 20:46:45.666061945 +0000 UTC m=+22.537165413" Jul 29 20:46:45.666638 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:45.666596 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l4txq" podStartSLOduration=3.921558306 podStartE2EDuration="22.666583954s" totalImagesPullingTime="18.745025648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:45.665322256 +0000 UTC m=+22.536425748" watchObservedRunningTime="2026-07-29 20:46:45.666583954 +0000 UTC m=+22.537687423" Jul 29 20:46:45.677991 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:45.677933 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mc9jj" podStartSLOduration=3.978939124 podStartE2EDuration="22.677902603s" totalImagesPullingTime="18.698963479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:45.677221372 +0000 UTC m=+22.548324839" watchObservedRunningTime="2026-07-29 20:46:45.677902603 +0000 UTC m=+22.549006069" Jul 29 20:46:45.691544 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:45.691495 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p87w8" podStartSLOduration=3.945749938 podStartE2EDuration="22.691456482s" totalImagesPullingTime="18.745706544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:45.691104966 +0000 UTC m=+22.562208435" watchObservedRunningTime="2026-07-29 20:46:45.691456482 +0000 UTC m=+22.562559948" Jul 29 20:46:46.551737 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:46.551570 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:46.552321 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:46.551806 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jd4g" podUID="4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5" Jul 29 20:46:46.661516 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:46.661443 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" event={"ID":"946ea891-4ad4-4279-843b-c93a2aee2164","Type":"ContainerStarted","Data":"bbee92cc6e7e89b636d9f668562819a0d4ff176c60d524ec396bbeca1f596331"} Jul 29 20:46:46.661516 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:46.661512 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" event={"ID":"946ea891-4ad4-4279-843b-c93a2aee2164","Type":"ContainerStarted","Data":"8790a8892874e0d4fde60b139602cfe689b63c33099540b65e9ce21d47828216"} Jul 29 20:46:46.661772 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:46.661550 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" event={"ID":"946ea891-4ad4-4279-843b-c93a2aee2164","Type":"ContainerStarted","Data":"cf17b042a6a7cadead54b23c942a1e974de9261b2122f38145b7c52ed58fc842"} Jul 29 20:46:46.661772 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:46.661584 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" event={"ID":"946ea891-4ad4-4279-843b-c93a2aee2164","Type":"ContainerStarted","Data":"9af083dc8f4ff66c49714208d42649ae6e0022f94135aef6817b83d9a44efed3"} Jul 29 20:46:46.661772 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:46.661614 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" event={"ID":"946ea891-4ad4-4279-843b-c93a2aee2164","Type":"ContainerStarted","Data":"00ff61ee7ccfff5144dfa2199898631532afc32bce7b3fdf808f781697c6068d"} Jul 29 20:46:46.661772 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:46.661652 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" event={"ID":"946ea891-4ad4-4279-843b-c93a2aee2164","Type":"ContainerStarted","Data":"00c465314cafee2684620cf6485ec96afc1bb66f494fde1f391a85582db4e4cd"} Jul 29 20:46:46.662745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:46.662722 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dc84m" event={"ID":"744f3fb4-81ae-4b9c-9bc3-7fb975c91336","Type":"ContainerStarted","Data":"08374b9ae949f79e5b68607aa196184dc583d82b538126aae7ab5070ec4d8719"} Jul 29 20:46:46.663956 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:46.663911 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-njm9k" event={"ID":"b1661ddb-eba2-4408-a48b-f2edcc9b600e","Type":"ContainerStarted","Data":"28cf6750a95a2d342c7019c38c8bb6543a44d268d9ca12e18fb0c5db3bc24510"} Jul 29 20:46:46.667949 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:46.667904 2562 generic.go:356] "Generic (PLEG): container finished" podID="a11939fc-63a4-4f61-9de5-5d9bef849715" containerID="0f7d65c2feade383244abfa4f701315d91cd6cb690566d76d44e21d581a4f8dd" exitCode=0 Jul 29 20:46:46.668049 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:46.668010 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9nwx" event={"ID":"a11939fc-63a4-4f61-9de5-5d9bef849715","Type":"ContainerDied","Data":"0f7d65c2feade383244abfa4f701315d91cd6cb690566d76d44e21d581a4f8dd"} Jul 29 20:46:46.677826 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:46.677785 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dc84m" podStartSLOduration=4.934335233 podStartE2EDuration="23.677773972s" totalImagesPullingTime="18.743438739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:46.677476679 +0000 UTC m=+23.548580146" watchObservedRunningTime="2026-07-29 20:46:46.677773972 +0000 UTC m=+23.548877436" Jul 29 20:46:46.697541 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:46.697507 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-njm9k" podStartSLOduration=4.553384257 podStartE2EDuration="23.697494997s" totalImagesPullingTime="19.14411074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:46.69701112 +0000 UTC m=+23.568114588" watchObservedRunningTime="2026-07-29 20:46:46.697494997 +0000 UTC m=+23.568598464" Jul 29 20:46:46.934860 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:46.934838 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 20:46:47.529435 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:47.529328 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T20:46:46.934854504Z","UUID":"334ce7a3-79f6-47ca-a09a-aa968275f8e9","Handler":null,"Name":"","Endpoint":""} Jul 29 20:46:47.531562 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:47.531432 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 20:46:47.531562 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:47.531478 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 20:46:47.551179 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:47.551007 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:47.551179 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:47.551026 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:47.551456 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:47.551184 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhgt5" podUID="ba4aa585-620a-4c88-8c46-09b343186c70" Jul 29 20:46:47.551456 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:47.551315 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sgzv" podUID="347e06fd-9f35-47dd-9442-ddd628e5bd35" Jul 29 20:46:47.672451 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:47.672417 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" event={"ID":"04580c8c-3311-478b-b918-debc86d16f14","Type":"ContainerStarted","Data":"f4fdd494fa41a8cc77294f080a8bb1c55ed38b1797b46f941887a87ec0ae3a62"} Jul 29 20:46:47.674080 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:47.674053 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5tkhm" event={"ID":"0f0a3ba3-928b-431b-a554-78a8ecb072f6","Type":"ContainerStarted","Data":"b9f4238b6abb692bd3ae01cbddc5489404dd2d590bd3590c36b479ebdbbbc934"} Jul 29 20:46:47.687748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:47.687707 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5tkhm" podStartSLOduration=4.939817007 podStartE2EDuration="23.687693176s" totalImagesPullingTime="18.747876169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:47.68730594 +0000 UTC m=+24.558409408" watchObservedRunningTime="2026-07-29 20:46:47.687693176 +0000 UTC m=+24.558796644" Jul 29 20:46:48.132931 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:48.132895 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mc9jj" Jul 29 20:46:48.218630 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:48.218604 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mc9jj" Jul 29 20:46:48.219320 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:48.219298 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mc9jj" Jul 29 20:46:48.551096 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:48.550880 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:48.551266 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:48.551194 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jd4g" podUID="4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5" Jul 29 20:46:48.679668 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:48.679576 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" event={"ID":"946ea891-4ad4-4279-843b-c93a2aee2164","Type":"ContainerStarted","Data":"a44a6db719cfdebd530be4babc355187ce205719e840347d795d5a4a0f1dac40"} Jul 29 20:46:48.681937 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:48.681887 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" event={"ID":"04580c8c-3311-478b-b918-debc86d16f14","Type":"ContainerStarted","Data":"157d6bab705a5044167ef49501d8f130d607fd07ddee673307b6b54344e19eee"} Jul 29 20:46:48.682623 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:48.682602 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mc9jj" Jul 29 20:46:48.699745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:48.699676 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4db4" podStartSLOduration=4.672244804 podStartE2EDuration="25.699662286s" totalImagesPullingTime="21.027417482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:48.699183225 +0000 UTC m=+25.570286692" watchObservedRunningTime="2026-07-29 20:46:48.699662286 +0000 UTC m=+25.570765752" Jul 29 20:46:49.550741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:49.550704 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:49.550741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:49.550742 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:49.551022 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:49.550874 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sgzv" podUID="347e06fd-9f35-47dd-9442-ddd628e5bd35" Jul 29 20:46:49.551022 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:49.550990 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhgt5" podUID="ba4aa585-620a-4c88-8c46-09b343186c70" Jul 29 20:46:50.550914 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:50.550877 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:50.551505 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:50.551015 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jd4g" podUID="4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5" Jul 29 20:46:51.550984 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:51.550777 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:51.551794 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:51.550777 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:51.551794 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:51.551063 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sgzv" podUID="347e06fd-9f35-47dd-9442-ddd628e5bd35" Jul 29 20:46:51.551794 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:51.551158 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhgt5" podUID="ba4aa585-620a-4c88-8c46-09b343186c70" Jul 29 20:46:51.690609 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:51.690535 2562 generic.go:356] "Generic (PLEG): container finished" podID="a11939fc-63a4-4f61-9de5-5d9bef849715" containerID="7eba7165d34e7bec02bb7555f5b6762e9dcae34a825a2220b5210ce597aadde7" exitCode=0 Jul 29 20:46:51.690774 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:51.690624 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9nwx" event={"ID":"a11939fc-63a4-4f61-9de5-5d9bef849715","Type":"ContainerDied","Data":"7eba7165d34e7bec02bb7555f5b6762e9dcae34a825a2220b5210ce597aadde7"} Jul 29 20:46:51.694380 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:51.694357 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" event={"ID":"946ea891-4ad4-4279-843b-c93a2aee2164","Type":"ContainerStarted","Data":"681c4a9947473b8ad334e76fd05cf54a39a3240354fca74df6db7217943d9de8"} Jul 29 20:46:51.694713 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:51.694691 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:51.694813 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:51.694750 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:51.694813 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:51.694797 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:51.709736 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:51.709716 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:51.710017 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:51.709954 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:46:51.736207 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:51.736163 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" podStartSLOduration=9.970649745 podStartE2EDuration="28.736149967s" totalImagesPullingTime="18.765500222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:46:51.735575286 +0000 UTC m=+28.606678753" watchObservedRunningTime="2026-07-29 20:46:51.736149967 +0000 UTC m=+28.607253433" Jul 29 20:46:52.551168 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:52.551144 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:52.551726 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:52.551265 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jd4g" podUID="4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5" Jul 29 20:46:53.355561 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:53.355380 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fhgt5"] Jul 29 20:46:53.355720 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:53.355638 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:53.355786 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:53.355722 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhgt5" podUID="ba4aa585-620a-4c88-8c46-09b343186c70" Jul 29 20:46:53.359112 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:53.359083 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7jd4g"] Jul 29 20:46:53.359235 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:53.359178 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:53.359305 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:53.359284 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jd4g" podUID="4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5" Jul 29 20:46:53.359957 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:53.359932 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5sgzv"] Jul 29 20:46:53.360063 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:53.360043 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:53.360177 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:53.360143 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sgzv" podUID="347e06fd-9f35-47dd-9442-ddd628e5bd35" Jul 29 20:46:53.700644 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:53.700610 2562 generic.go:356] "Generic (PLEG): container finished" podID="a11939fc-63a4-4f61-9de5-5d9bef849715" containerID="ccd9dc4681e7f0b1e673ef94ce88af36738c088e21972ac236d1b90fb29476e8" exitCode=0 Jul 29 20:46:53.701007 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:53.700670 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9nwx" event={"ID":"a11939fc-63a4-4f61-9de5-5d9bef849715","Type":"ContainerDied","Data":"ccd9dc4681e7f0b1e673ef94ce88af36738c088e21972ac236d1b90fb29476e8"} Jul 29 20:46:54.551229 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:54.551205 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:54.551334 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:54.551314 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jd4g" podUID="4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5" Jul 29 20:46:55.551421 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:55.551336 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:55.551421 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:55.551385 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:55.552078 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:55.551489 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sgzv" podUID="347e06fd-9f35-47dd-9442-ddd628e5bd35" Jul 29 20:46:55.552078 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:55.551618 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhgt5" podUID="ba4aa585-620a-4c88-8c46-09b343186c70" Jul 29 20:46:55.707354 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:55.707324 2562 generic.go:356] "Generic (PLEG): container finished" podID="a11939fc-63a4-4f61-9de5-5d9bef849715" containerID="a860c50d832e9701ce7cfa066b01b83064b8881a7f9a598d7baaf3fef21da096" exitCode=0 Jul 29 20:46:55.707491 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:55.707382 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9nwx" event={"ID":"a11939fc-63a4-4f61-9de5-5d9bef849715","Type":"ContainerDied","Data":"a860c50d832e9701ce7cfa066b01b83064b8881a7f9a598d7baaf3fef21da096"} Jul 29 20:46:56.551410 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:56.551378 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:56.551586 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:56.551480 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7jd4g" podUID="4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5" Jul 29 20:46:57.218237 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:57.218160 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret\") pod \"global-pull-secret-syncer-fhgt5\" (UID: \"ba4aa585-620a-4c88-8c46-09b343186c70\") " pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:57.218237 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:57.218208 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs\") pod \"network-metrics-daemon-5sgzv\" (UID: \"347e06fd-9f35-47dd-9442-ddd628e5bd35\") " pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:57.218434 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:57.218317 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 20:46:57.218434 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:57.218346 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:46:57.218434 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:57.218406 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret podName:ba4aa585-620a-4c88-8c46-09b343186c70 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:29.21838881 +0000 UTC m=+66.089492264 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret") pod "global-pull-secret-syncer-fhgt5" (UID: "ba4aa585-620a-4c88-8c46-09b343186c70") : object "kube-system"/"original-pull-secret" not registered Jul 29 20:46:57.218434 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:57.218423 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs podName:347e06fd-9f35-47dd-9442-ddd628e5bd35 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:29.218416697 +0000 UTC m=+66.089520142 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs") pod "network-metrics-daemon-5sgzv" (UID: "347e06fd-9f35-47dd-9442-ddd628e5bd35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 20:46:57.319556 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:57.319520 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m98sf\" (UniqueName: \"kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf\") pod \"network-check-target-7jd4g\" (UID: \"4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5\") " pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:57.319717 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:57.319654 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 20:46:57.319717 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:57.319675 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 20:46:57.319717 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:57.319684 2562 projected.go:196] Error preparing data for projected volume kube-api-access-m98sf for pod openshift-network-diagnostics/network-check-target-7jd4g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:46:57.319865 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:57.319742 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf podName:4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:29.319722143 +0000 UTC m=+66.190825604 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m98sf" (UniqueName: "kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf") pod "network-check-target-7jd4g" (UID: "4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 20:46:57.551914 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:57.551383 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:57.551914 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:57.551520 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sgzv" podUID="347e06fd-9f35-47dd-9442-ddd628e5bd35" Jul 29 20:46:57.551914 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:57.551583 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:57.551914 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:57.551705 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhgt5" podUID="ba4aa585-620a-4c88-8c46-09b343186c70" Jul 29 20:46:58.427385 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.427353 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-216.ec2.internal" event="NodeReady" Jul 29 20:46:58.427546 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.427487 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 20:46:58.460727 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.460696 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f68896cdc-qpf7g"] Jul 29 20:46:58.463232 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.463209 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.465378 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.465344 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 29 20:46:58.465480 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.465399 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 29 20:46:58.465537 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.465497 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wf87m\"" Jul 29 20:46:58.465596 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.465344 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 29 20:46:58.470470 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.470366 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q9s7m"] Jul 29 20:46:58.471398 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.471196 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 29 20:46:58.472453 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.472431 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx"] Jul 29 20:46:58.472908 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.472645 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q9s7m" Jul 29 20:46:58.474129 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.474110 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5f46dc85cf-9562c"] Jul 29 20:46:58.474256 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.474235 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" Jul 29 20:46:58.475367 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.475329 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 29 20:46:58.475460 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.475367 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fxg9q\"" Jul 29 20:46:58.475514 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.475475 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 29 20:46:58.476683 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.476504 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 29 20:46:58.480056 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.477186 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 20:46:58.480056 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.477399 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-p4wrc\"" Jul 29 20:46:58.480056 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.477609 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:46:58.480056 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.477697 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 20:46:58.481408 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.481212 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 29 20:46:58.481408 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.481257 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 29 20:46:58.482051 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.482032 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cxpqv\"" Jul 29 20:46:58.482219 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.482195 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-8v2vf"] Jul 29 20:46:58.482431 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.482414 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 29 20:46:58.482611 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.482592 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 29 20:46:58.482674 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.482638 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 20:46:58.482758 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.482421 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 29 20:46:58.482937 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.482896 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 20:46:58.484118 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.484098 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-9hbmn"] Jul 29 20:46:58.484243 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.484223 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:46:58.486241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.486090 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 29 20:46:58.486241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.486136 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f68896cdc-qpf7g"] Jul 29 20:46:58.486241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.486149 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 29 20:46:58.486241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.486226 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.486754 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.486735 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 29 20:46:58.486843 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.486832 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-vbttn\"" Jul 29 20:46:58.487012 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.486994 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 29 20:46:58.489192 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.489030 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 29 20:46:58.489192 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.489048 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 20:46:58.489413 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.489268 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 29 20:46:58.489521 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.489505 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-h6nrv\"" Jul 29 20:46:58.490762 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.490743 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx"] Jul 29 20:46:58.490897 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.490883 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 20:46:58.492542 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.492225 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q9s7m"] Jul 29 20:46:58.494185 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.494158 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f46dc85cf-9562c"] Jul 29 20:46:58.494302 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.494203 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7"] Jul 29 20:46:58.496937 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.496900 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-q7z24"] Jul 29 20:46:58.497623 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.497605 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7" Jul 29 20:46:58.499520 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.499500 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 29 20:46:58.499861 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.499842 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 29 20:46:58.500303 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.500276 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-svzcn\"" Jul 29 20:46:58.500433 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.500417 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 29 20:46:58.500776 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.500746 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 29 20:46:58.500854 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.500837 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 29 20:46:58.501843 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.501822 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-t89sj"] Jul 29 20:46:58.501961 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.501940 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-q7z24" Jul 29 20:46:58.505416 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.505393 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-gncct\"" Jul 29 20:46:58.506584 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.506294 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 29 20:46:58.506783 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.506766 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 29 20:46:58.508899 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.508878 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vmz8d"] Jul 29 20:46:58.509006 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.508912 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 29 20:46:58.509006 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.508972 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t89sj" Jul 29 20:46:58.519387 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.511768 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zwgsv"] Jul 29 20:46:58.519387 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.511964 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vmz8d" Jul 29 20:46:58.519387 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.513247 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-79k6t\"" Jul 29 20:46:58.519387 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.513448 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 20:46:58.519387 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.513662 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 20:46:58.519387 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.516080 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-xrtjv"] Jul 29 20:46:58.519387 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.516194 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zwgsv" Jul 29 20:46:58.519387 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.518847 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-t5k6j\"" Jul 29 20:46:58.519387 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.519074 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 29 20:46:58.519387 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.519247 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 29 20:46:58.520016 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.519438 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 29 20:46:58.520016 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.519608 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 20:46:58.522606 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.520191 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 29 20:46:58.522606 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.520552 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cc48x\"" Jul 29 20:46:58.522606 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.520577 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 20:46:58.522606 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.521088 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v"] Jul 29 20:46:58.522606 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.521252 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xrtjv" Jul 29 20:46:58.523808 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.523105 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-9hbmn"] Jul 29 20:46:58.523808 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.523135 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-8v2vf"] Jul 29 20:46:58.523808 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.523161 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-q7z24"] Jul 29 20:46:58.523808 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.523181 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vmz8d"] Jul 29 20:46:58.523808 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.523196 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zwgsv"] Jul 29 20:46:58.523808 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.523207 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t89sj"] Jul 29 20:46:58.523808 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.523221 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7"] Jul 29 20:46:58.523808 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.523221 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v" Jul 29 20:46:58.523808 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.523233 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-xrtjv"] Jul 29 20:46:58.523808 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.523248 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v"] Jul 29 20:46:58.524692 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.524672 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 20:46:58.524943 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.524913 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 20:46:58.525315 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.525298 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 20:46:58.525523 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.525511 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-flvjv\"" Jul 29 20:46:58.525677 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.525662 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 29 20:46:58.525828 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.525814 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 29 20:46:58.526115 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.526099 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wmpmw\"" Jul 29 20:46:58.526263 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.526251 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 29 20:46:58.551202 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.551185 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:46:58.555138 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.555120 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zcmz5\"" Jul 29 20:46:58.628143 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628110 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw2n2\" (UniqueName: \"kubernetes.io/projected/a08729b6-2792-466e-b7c4-fb0ae44087e6-kube-api-access-dw2n2\") pod \"insights-operator-5885fdb759-9hbmn\" (UID: \"a08729b6-2792-466e-b7c4-fb0ae44087e6\") " pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.628301 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628154 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4250508f-31ea-4575-b84e-50a57d1aa159-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7\" (UID: \"4250508f-31ea-4575-b84e-50a57d1aa159\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7" Jul 29 20:46:58.628301 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628203 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30987044-0cc1-4e94-91d9-4712292608be-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-q7z24\" (UID: \"30987044-0cc1-4e94-91d9-4712292608be\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-q7z24" Jul 29 20:46:58.628301 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628233 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1f181cb2-5f41-425b-97d5-6057df483f86-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-brnnx\" (UID: \"1f181cb2-5f41-425b-97d5-6057df483f86\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" Jul 29 20:46:58.628301 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628261 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/863cac4c-638f-4789-88d9-263045a9c8ac-config\") pod \"service-ca-operator-5d6c75f84f-vmz8d\" (UID: \"863cac4c-638f-4789-88d9-263045a9c8ac\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vmz8d" Jul 29 20:46:58.628301 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628277 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.628301 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628292 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-stats-auth\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:46:58.628608 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628310 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc90f71b-6647-4bb2-af21-cfaf42f51393-image-registry-private-configuration\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.628608 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628333 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a08729b6-2792-466e-b7c4-fb0ae44087e6-service-ca-bundle\") pod \"insights-operator-5885fdb759-9hbmn\" (UID: \"a08729b6-2792-466e-b7c4-fb0ae44087e6\") " pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.628608 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628348 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx6r7\" (UniqueName: \"kubernetes.io/projected/863cac4c-638f-4789-88d9-263045a9c8ac-kube-api-access-qx6r7\") pod \"service-ca-operator-5d6c75f84f-vmz8d\" (UID: \"863cac4c-638f-4789-88d9-263045a9c8ac\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vmz8d" Jul 29 20:46:58.628608 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628366 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-nmw5v\" (UID: \"ffefc971-2b69-4967-bbd3-35da515c68df\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v" Jul 29 20:46:58.628608 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628383 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-brnnx\" (UID: \"1f181cb2-5f41-425b-97d5-6057df483f86\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" Jul 29 20:46:58.628608 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628404 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5b5n\" (UniqueName: \"kubernetes.io/projected/c4622d90-8ccb-4ee8-80a8-6a0a0d4e552d-kube-api-access-p5b5n\") pod \"volume-data-source-validator-549c6d54cf-q9s7m\" (UID: \"c4622d90-8ccb-4ee8-80a8-6a0a0d4e552d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q9s7m" Jul 29 20:46:58.628608 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628471 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a08729b6-2792-466e-b7c4-fb0ae44087e6-tmp\") pod \"insights-operator-5885fdb759-9hbmn\" (UID: \"a08729b6-2792-466e-b7c4-fb0ae44087e6\") " pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.628608 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628518 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4250508f-31ea-4575-b84e-50a57d1aa159-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7\" (UID: \"4250508f-31ea-4575-b84e-50a57d1aa159\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7" Jul 29 20:46:58.628608 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628543 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkhh5\" (UniqueName: \"kubernetes.io/projected/41604e7f-766a-4d75-853d-371e7f7e6a92-kube-api-access-kkhh5\") pod \"network-check-source-7d88cb55d8-xrtjv\" (UID: \"41604e7f-766a-4d75-853d-371e7f7e6a92\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xrtjv" Jul 29 20:46:58.628608 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628569 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf9e0545-985e-47df-af42-4453c1e24937-trusted-ca\") pod \"console-operator-584dcb6b54-8v2vf\" (UID: \"bf9e0545-985e-47df-af42-4453c1e24937\") " pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:46:58.628608 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628600 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9db9c\" (UniqueName: \"kubernetes.io/projected/ffefc971-2b69-4967-bbd3-35da515c68df-kube-api-access-9db9c\") pod \"cluster-samples-operator-6748489656-nmw5v\" (UID: \"ffefc971-2b69-4967-bbd3-35da515c68df\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v" Jul 29 20:46:58.629075 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628622 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:46:58.629075 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628684 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-certificates\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.629075 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628710 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5cws\" (UniqueName: \"kubernetes.io/projected/9749b441-c685-4fa6-8044-791cae15b712-kube-api-access-l5cws\") pod \"dns-default-t89sj\" (UID: \"9749b441-c685-4fa6-8044-791cae15b712\") " pod="openshift-dns/dns-default-t89sj" Jul 29 20:46:58.629075 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628735 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc90f71b-6647-4bb2-af21-cfaf42f51393-installation-pull-secrets\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.629075 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628788 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9749b441-c685-4fa6-8044-791cae15b712-tmp-dir\") pod \"dns-default-t89sj\" (UID: \"9749b441-c685-4fa6-8044-791cae15b712\") " pod="openshift-dns/dns-default-t89sj" Jul 29 20:46:58.629075 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628813 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-q7z24\" (UID: \"30987044-0cc1-4e94-91d9-4712292608be\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-q7z24" Jul 29 20:46:58.629075 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628835 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a08729b6-2792-466e-b7c4-fb0ae44087e6-snapshots\") pod \"insights-operator-5885fdb759-9hbmn\" (UID: \"a08729b6-2792-466e-b7c4-fb0ae44087e6\") " pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.629075 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628851 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:46:58.629075 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628902 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwqbq\" (UniqueName: \"kubernetes.io/projected/bf9e0545-985e-47df-af42-4453c1e24937-kube-api-access-mwqbq\") pod \"console-operator-584dcb6b54-8v2vf\" (UID: \"bf9e0545-985e-47df-af42-4453c1e24937\") " pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:46:58.629075 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628933 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/863cac4c-638f-4789-88d9-263045a9c8ac-serving-cert\") pod \"service-ca-operator-5d6c75f84f-vmz8d\" (UID: \"863cac4c-638f-4789-88d9-263045a9c8ac\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vmz8d" Jul 29 20:46:58.629075 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628972 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-bound-sa-token\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.629075 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.628999 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a08729b6-2792-466e-b7c4-fb0ae44087e6-serving-cert\") pod \"insights-operator-5885fdb759-9hbmn\" (UID: \"a08729b6-2792-466e-b7c4-fb0ae44087e6\") " pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.629075 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.629021 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lngx6\" (UniqueName: \"kubernetes.io/projected/4250508f-31ea-4575-b84e-50a57d1aa159-kube-api-access-lngx6\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7\" (UID: \"4250508f-31ea-4575-b84e-50a57d1aa159\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7" Jul 29 20:46:58.629075 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.629044 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxgj4\" (UniqueName: \"kubernetes.io/projected/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-kube-api-access-zxgj4\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:46:58.629075 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.629079 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9749b441-c685-4fa6-8044-791cae15b712-config-volume\") pod \"dns-default-t89sj\" (UID: \"9749b441-c685-4fa6-8044-791cae15b712\") " pod="openshift-dns/dns-default-t89sj" Jul 29 20:46:58.629657 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.629103 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls\") pod \"dns-default-t89sj\" (UID: \"9749b441-c685-4fa6-8044-791cae15b712\") " pod="openshift-dns/dns-default-t89sj" Jul 29 20:46:58.629657 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.629119 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a08729b6-2792-466e-b7c4-fb0ae44087e6-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-9hbmn\" (UID: \"a08729b6-2792-466e-b7c4-fb0ae44087e6\") " pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.629657 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.629135 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf9e0545-985e-47df-af42-4453c1e24937-serving-cert\") pod \"console-operator-584dcb6b54-8v2vf\" (UID: \"bf9e0545-985e-47df-af42-4453c1e24937\") " pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:46:58.629657 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.629153 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxg2d\" (UniqueName: \"kubernetes.io/projected/43c0ba7b-1895-4952-b45b-87f2983002d1-kube-api-access-kxg2d\") pod \"ingress-canary-zwgsv\" (UID: \"43c0ba7b-1895-4952-b45b-87f2983002d1\") " pod="openshift-ingress-canary/ingress-canary-zwgsv" Jul 29 20:46:58.629657 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.629201 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43c0ba7b-1895-4952-b45b-87f2983002d1-cert\") pod \"ingress-canary-zwgsv\" (UID: \"43c0ba7b-1895-4952-b45b-87f2983002d1\") " pod="openshift-ingress-canary/ingress-canary-zwgsv" Jul 29 20:46:58.629657 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.629235 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-default-certificate\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:46:58.629657 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.629282 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc90f71b-6647-4bb2-af21-cfaf42f51393-ca-trust-extracted\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.629657 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.629330 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf9e0545-985e-47df-af42-4453c1e24937-config\") pod \"console-operator-584dcb6b54-8v2vf\" (UID: \"bf9e0545-985e-47df-af42-4453c1e24937\") " pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:46:58.629657 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.629358 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc90f71b-6647-4bb2-af21-cfaf42f51393-trusted-ca\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.629657 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.629400 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sztj7\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-kube-api-access-sztj7\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.629657 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.629442 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgf74\" (UniqueName: \"kubernetes.io/projected/1f181cb2-5f41-425b-97d5-6057df483f86-kube-api-access-sgf74\") pod \"cluster-monitoring-operator-548b489cb8-brnnx\" (UID: \"1f181cb2-5f41-425b-97d5-6057df483f86\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" Jul 29 20:46:58.730676 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.730589 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9749b441-c685-4fa6-8044-791cae15b712-tmp-dir\") pod \"dns-default-t89sj\" (UID: \"9749b441-c685-4fa6-8044-791cae15b712\") " pod="openshift-dns/dns-default-t89sj" Jul 29 20:46:58.730676 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.730634 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-q7z24\" (UID: \"30987044-0cc1-4e94-91d9-4712292608be\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-q7z24" Jul 29 20:46:58.730676 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.730663 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a08729b6-2792-466e-b7c4-fb0ae44087e6-snapshots\") pod \"insights-operator-5885fdb759-9hbmn\" (UID: \"a08729b6-2792-466e-b7c4-fb0ae44087e6\") " pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.730970 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.730686 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:46:58.730970 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.730710 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwqbq\" (UniqueName: \"kubernetes.io/projected/bf9e0545-985e-47df-af42-4453c1e24937-kube-api-access-mwqbq\") pod \"console-operator-584dcb6b54-8v2vf\" (UID: \"bf9e0545-985e-47df-af42-4453c1e24937\") " pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:46:58.730970 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.730728 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/863cac4c-638f-4789-88d9-263045a9c8ac-serving-cert\") pod \"service-ca-operator-5d6c75f84f-vmz8d\" (UID: \"863cac4c-638f-4789-88d9-263045a9c8ac\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vmz8d" Jul 29 20:46:58.730970 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.730747 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-bound-sa-token\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.730970 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.730769 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a08729b6-2792-466e-b7c4-fb0ae44087e6-serving-cert\") pod \"insights-operator-5885fdb759-9hbmn\" (UID: \"a08729b6-2792-466e-b7c4-fb0ae44087e6\") " pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.730970 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.730789 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lngx6\" (UniqueName: \"kubernetes.io/projected/4250508f-31ea-4575-b84e-50a57d1aa159-kube-api-access-lngx6\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7\" (UID: \"4250508f-31ea-4575-b84e-50a57d1aa159\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7" Jul 29 20:46:58.730970 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:58.730801 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 20:46:58.730970 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:58.730960 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert podName:30987044-0cc1-4e94-91d9-4712292608be nodeName:}" failed. No retries permitted until 2026-07-29 20:46:59.23093903 +0000 UTC m=+36.102042475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-q7z24" (UID: "30987044-0cc1-4e94-91d9-4712292608be") : secret "networking-console-plugin-cert" not found Jul 29 20:46:58.730970 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.730804 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxgj4\" (UniqueName: \"kubernetes.io/projected/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-kube-api-access-zxgj4\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:46:58.731396 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.731010 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9749b441-c685-4fa6-8044-791cae15b712-config-volume\") pod \"dns-default-t89sj\" (UID: \"9749b441-c685-4fa6-8044-791cae15b712\") " pod="openshift-dns/dns-default-t89sj" Jul 29 20:46:58.731396 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.731036 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls\") pod \"dns-default-t89sj\" (UID: \"9749b441-c685-4fa6-8044-791cae15b712\") " pod="openshift-dns/dns-default-t89sj" Jul 29 20:46:58.731396 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.731053 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9749b441-c685-4fa6-8044-791cae15b712-tmp-dir\") pod \"dns-default-t89sj\" (UID: \"9749b441-c685-4fa6-8044-791cae15b712\") " pod="openshift-dns/dns-default-t89sj" Jul 29 20:46:58.731396 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.731061 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a08729b6-2792-466e-b7c4-fb0ae44087e6-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-9hbmn\" (UID: \"a08729b6-2792-466e-b7c4-fb0ae44087e6\") " pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.731396 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.731090 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf9e0545-985e-47df-af42-4453c1e24937-serving-cert\") pod \"console-operator-584dcb6b54-8v2vf\" (UID: \"bf9e0545-985e-47df-af42-4453c1e24937\") " pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:46:58.731396 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.731116 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxg2d\" (UniqueName: \"kubernetes.io/projected/43c0ba7b-1895-4952-b45b-87f2983002d1-kube-api-access-kxg2d\") pod \"ingress-canary-zwgsv\" (UID: \"43c0ba7b-1895-4952-b45b-87f2983002d1\") " pod="openshift-ingress-canary/ingress-canary-zwgsv" Jul 29 20:46:58.731396 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.731141 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43c0ba7b-1895-4952-b45b-87f2983002d1-cert\") pod \"ingress-canary-zwgsv\" (UID: \"43c0ba7b-1895-4952-b45b-87f2983002d1\") " pod="openshift-ingress-canary/ingress-canary-zwgsv" Jul 29 20:46:58.731396 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.731164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-default-certificate\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:46:58.731396 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.731192 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc90f71b-6647-4bb2-af21-cfaf42f51393-ca-trust-extracted\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.731396 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.731228 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf9e0545-985e-47df-af42-4453c1e24937-config\") pod \"console-operator-584dcb6b54-8v2vf\" (UID: \"bf9e0545-985e-47df-af42-4453c1e24937\") " pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:46:58.731396 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.731247 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc90f71b-6647-4bb2-af21-cfaf42f51393-trusted-ca\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.731396 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.731267 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sztj7\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-kube-api-access-sztj7\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.731396 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.731304 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgf74\" (UniqueName: \"kubernetes.io/projected/1f181cb2-5f41-425b-97d5-6057df483f86-kube-api-access-sgf74\") pod \"cluster-monitoring-operator-548b489cb8-brnnx\" (UID: \"1f181cb2-5f41-425b-97d5-6057df483f86\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" Jul 29 20:46:58.731396 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:58.731330 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle podName:d56bff06-5480-4bcc-bf5b-dbfd2372f9cd nodeName:}" failed. No retries permitted until 2026-07-29 20:46:59.231308871 +0000 UTC m=+36.102412335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle") pod "router-default-5f46dc85cf-9562c" (UID: "d56bff06-5480-4bcc-bf5b-dbfd2372f9cd") : configmap references non-existent config key: service-ca.crt Jul 29 20:46:58.731396 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.731374 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw2n2\" (UniqueName: \"kubernetes.io/projected/a08729b6-2792-466e-b7c4-fb0ae44087e6-kube-api-access-dw2n2\") pod \"insights-operator-5885fdb759-9hbmn\" (UID: \"a08729b6-2792-466e-b7c4-fb0ae44087e6\") " pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.731396 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.731398 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a08729b6-2792-466e-b7c4-fb0ae44087e6-snapshots\") pod \"insights-operator-5885fdb759-9hbmn\" (UID: \"a08729b6-2792-466e-b7c4-fb0ae44087e6\") " pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.732178 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.731408 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4250508f-31ea-4575-b84e-50a57d1aa159-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7\" (UID: \"4250508f-31ea-4575-b84e-50a57d1aa159\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7" Jul 29 20:46:58.732178 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.731436 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30987044-0cc1-4e94-91d9-4712292608be-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-q7z24\" (UID: \"30987044-0cc1-4e94-91d9-4712292608be\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-q7z24" Jul 29 20:46:58.732178 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.731463 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1f181cb2-5f41-425b-97d5-6057df483f86-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-brnnx\" (UID: \"1f181cb2-5f41-425b-97d5-6057df483f86\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" Jul 29 20:46:58.732178 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.731987 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9749b441-c685-4fa6-8044-791cae15b712-config-volume\") pod \"dns-default-t89sj\" (UID: \"9749b441-c685-4fa6-8044-791cae15b712\") " pod="openshift-dns/dns-default-t89sj" Jul 29 20:46:58.732374 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.732269 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1f181cb2-5f41-425b-97d5-6057df483f86-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-brnnx\" (UID: \"1f181cb2-5f41-425b-97d5-6057df483f86\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" Jul 29 20:46:58.732425 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.732375 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a08729b6-2792-466e-b7c4-fb0ae44087e6-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-9hbmn\" (UID: \"a08729b6-2792-466e-b7c4-fb0ae44087e6\") " pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.732425 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.732418 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/863cac4c-638f-4789-88d9-263045a9c8ac-config\") pod \"service-ca-operator-5d6c75f84f-vmz8d\" (UID: \"863cac4c-638f-4789-88d9-263045a9c8ac\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vmz8d" Jul 29 20:46:58.732528 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.732454 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.732528 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.732476 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-stats-auth\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:46:58.732528 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.732500 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc90f71b-6647-4bb2-af21-cfaf42f51393-image-registry-private-configuration\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.732674 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.732531 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a08729b6-2792-466e-b7c4-fb0ae44087e6-service-ca-bundle\") pod \"insights-operator-5885fdb759-9hbmn\" (UID: \"a08729b6-2792-466e-b7c4-fb0ae44087e6\") " pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.732674 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.732555 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx6r7\" (UniqueName: \"kubernetes.io/projected/863cac4c-638f-4789-88d9-263045a9c8ac-kube-api-access-qx6r7\") pod \"service-ca-operator-5d6c75f84f-vmz8d\" (UID: \"863cac4c-638f-4789-88d9-263045a9c8ac\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vmz8d" Jul 29 20:46:58.732674 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.732564 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30987044-0cc1-4e94-91d9-4712292608be-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-q7z24\" (UID: \"30987044-0cc1-4e94-91d9-4712292608be\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-q7z24" Jul 29 20:46:58.732674 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.732581 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-nmw5v\" (UID: \"ffefc971-2b69-4967-bbd3-35da515c68df\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v" Jul 29 20:46:58.732674 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:58.732657 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 20:46:58.732910 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:58.732705 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls podName:ffefc971-2b69-4967-bbd3-35da515c68df nodeName:}" failed. No retries permitted until 2026-07-29 20:46:59.23269246 +0000 UTC m=+36.103795905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls") pod "cluster-samples-operator-6748489656-nmw5v" (UID: "ffefc971-2b69-4967-bbd3-35da515c68df") : secret "samples-operator-tls" not found Jul 29 20:46:58.732910 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.732858 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf9e0545-985e-47df-af42-4453c1e24937-config\") pod \"console-operator-584dcb6b54-8v2vf\" (UID: \"bf9e0545-985e-47df-af42-4453c1e24937\") " pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:46:58.733247 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.733224 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/863cac4c-638f-4789-88d9-263045a9c8ac-config\") pod \"service-ca-operator-5d6c75f84f-vmz8d\" (UID: \"863cac4c-638f-4789-88d9-263045a9c8ac\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vmz8d" Jul 29 20:46:58.733247 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.733223 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc90f71b-6647-4bb2-af21-cfaf42f51393-ca-trust-extracted\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.733247 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.733238 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc90f71b-6647-4bb2-af21-cfaf42f51393-trusted-ca\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.733447 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:58.733314 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 20:46:58.733447 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:58.733317 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 20:46:58.733447 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:58.733328 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f68896cdc-qpf7g: secret "image-registry-tls" not found Jul 29 20:46:58.733447 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:58.733370 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls podName:cc90f71b-6647-4bb2-af21-cfaf42f51393 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:59.233354143 +0000 UTC m=+36.104457593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls") pod "image-registry-7f68896cdc-qpf7g" (UID: "cc90f71b-6647-4bb2-af21-cfaf42f51393") : secret "image-registry-tls" not found Jul 29 20:46:58.733447 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:58.733388 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls podName:9749b441-c685-4fa6-8044-791cae15b712 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:59.233378333 +0000 UTC m=+36.104481804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls") pod "dns-default-t89sj" (UID: "9749b441-c685-4fa6-8044-791cae15b712") : secret "dns-default-metrics-tls" not found Jul 29 20:46:58.734962 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.733849 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a08729b6-2792-466e-b7c4-fb0ae44087e6-service-ca-bundle\") pod \"insights-operator-5885fdb759-9hbmn\" (UID: \"a08729b6-2792-466e-b7c4-fb0ae44087e6\") " pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.734962 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.734122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-brnnx\" (UID: \"1f181cb2-5f41-425b-97d5-6057df483f86\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" Jul 29 20:46:58.734962 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.734161 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5b5n\" (UniqueName: \"kubernetes.io/projected/c4622d90-8ccb-4ee8-80a8-6a0a0d4e552d-kube-api-access-p5b5n\") pod \"volume-data-source-validator-549c6d54cf-q9s7m\" (UID: \"c4622d90-8ccb-4ee8-80a8-6a0a0d4e552d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q9s7m" Jul 29 20:46:58.734962 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.734189 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a08729b6-2792-466e-b7c4-fb0ae44087e6-tmp\") pod \"insights-operator-5885fdb759-9hbmn\" (UID: \"a08729b6-2792-466e-b7c4-fb0ae44087e6\") " pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.734962 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.734216 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4250508f-31ea-4575-b84e-50a57d1aa159-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7\" (UID: \"4250508f-31ea-4575-b84e-50a57d1aa159\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7" Jul 29 20:46:58.734962 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.734249 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhh5\" (UniqueName: \"kubernetes.io/projected/41604e7f-766a-4d75-853d-371e7f7e6a92-kube-api-access-kkhh5\") pod \"network-check-source-7d88cb55d8-xrtjv\" (UID: \"41604e7f-766a-4d75-853d-371e7f7e6a92\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xrtjv" Jul 29 20:46:58.734962 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.734275 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf9e0545-985e-47df-af42-4453c1e24937-trusted-ca\") pod \"console-operator-584dcb6b54-8v2vf\" (UID: \"bf9e0545-985e-47df-af42-4453c1e24937\") " pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:46:58.734962 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.734306 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9db9c\" (UniqueName: \"kubernetes.io/projected/ffefc971-2b69-4967-bbd3-35da515c68df-kube-api-access-9db9c\") pod \"cluster-samples-operator-6748489656-nmw5v\" (UID: \"ffefc971-2b69-4967-bbd3-35da515c68df\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v" Jul 29 20:46:58.734962 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.734329 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:46:58.734962 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.734377 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-certificates\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.734962 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.734413 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5cws\" (UniqueName: \"kubernetes.io/projected/9749b441-c685-4fa6-8044-791cae15b712-kube-api-access-l5cws\") pod \"dns-default-t89sj\" (UID: \"9749b441-c685-4fa6-8044-791cae15b712\") " pod="openshift-dns/dns-default-t89sj" Jul 29 20:46:58.734962 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.734436 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc90f71b-6647-4bb2-af21-cfaf42f51393-installation-pull-secrets\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.734962 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.734605 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4250508f-31ea-4575-b84e-50a57d1aa159-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7\" (UID: \"4250508f-31ea-4575-b84e-50a57d1aa159\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7" Jul 29 20:46:58.734962 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.734936 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a08729b6-2792-466e-b7c4-fb0ae44087e6-tmp\") pod \"insights-operator-5885fdb759-9hbmn\" (UID: \"a08729b6-2792-466e-b7c4-fb0ae44087e6\") " pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.737237 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.736128 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf9e0545-985e-47df-af42-4453c1e24937-trusted-ca\") pod \"console-operator-584dcb6b54-8v2vf\" (UID: \"bf9e0545-985e-47df-af42-4453c1e24937\") " pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:46:58.737237 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:58.734278 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 20:46:58.737237 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:58.736211 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls podName:1f181cb2-5f41-425b-97d5-6057df483f86 nodeName:}" failed. No retries permitted until 2026-07-29 20:46:59.236196154 +0000 UTC m=+36.107299599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-brnnx" (UID: "1f181cb2-5f41-425b-97d5-6057df483f86") : secret "cluster-monitoring-operator-tls" not found Jul 29 20:46:58.737237 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.736676 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-certificates\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.737237 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.736727 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-stats-auth\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:46:58.737237 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:58.736759 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 20:46:58.737237 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:58.736797 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs podName:d56bff06-5480-4bcc-bf5b-dbfd2372f9cd nodeName:}" failed. No retries permitted until 2026-07-29 20:46:59.236783836 +0000 UTC m=+36.107887283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs") pod "router-default-5f46dc85cf-9562c" (UID: "d56bff06-5480-4bcc-bf5b-dbfd2372f9cd") : secret "router-metrics-certs-default" not found Jul 29 20:46:58.737237 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.737204 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-default-certificate\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:46:58.737876 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.737849 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf9e0545-985e-47df-af42-4453c1e24937-serving-cert\") pod \"console-operator-584dcb6b54-8v2vf\" (UID: \"bf9e0545-985e-47df-af42-4453c1e24937\") " pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:46:58.738173 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.738148 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc90f71b-6647-4bb2-af21-cfaf42f51393-image-registry-private-configuration\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.738427 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.738350 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/863cac4c-638f-4789-88d9-263045a9c8ac-serving-cert\") pod \"service-ca-operator-5d6c75f84f-vmz8d\" (UID: \"863cac4c-638f-4789-88d9-263045a9c8ac\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vmz8d" Jul 29 20:46:58.738707 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.738684 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43c0ba7b-1895-4952-b45b-87f2983002d1-cert\") pod \"ingress-canary-zwgsv\" (UID: \"43c0ba7b-1895-4952-b45b-87f2983002d1\") " pod="openshift-ingress-canary/ingress-canary-zwgsv" Jul 29 20:46:58.739227 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.739207 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a08729b6-2792-466e-b7c4-fb0ae44087e6-serving-cert\") pod \"insights-operator-5885fdb759-9hbmn\" (UID: \"a08729b6-2792-466e-b7c4-fb0ae44087e6\") " pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.739316 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.739247 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4250508f-31ea-4575-b84e-50a57d1aa159-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7\" (UID: \"4250508f-31ea-4575-b84e-50a57d1aa159\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7" Jul 29 20:46:58.739443 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.739426 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc90f71b-6647-4bb2-af21-cfaf42f51393-installation-pull-secrets\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.743146 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.743126 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwqbq\" (UniqueName: \"kubernetes.io/projected/bf9e0545-985e-47df-af42-4453c1e24937-kube-api-access-mwqbq\") pod \"console-operator-584dcb6b54-8v2vf\" (UID: \"bf9e0545-985e-47df-af42-4453c1e24937\") " pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:46:58.744502 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.743980 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sztj7\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-kube-api-access-sztj7\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.744594 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.744535 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw2n2\" (UniqueName: \"kubernetes.io/projected/a08729b6-2792-466e-b7c4-fb0ae44087e6-kube-api-access-dw2n2\") pod \"insights-operator-5885fdb759-9hbmn\" (UID: \"a08729b6-2792-466e-b7c4-fb0ae44087e6\") " pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.744653 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.744597 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxgj4\" (UniqueName: \"kubernetes.io/projected/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-kube-api-access-zxgj4\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:46:58.745292 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.745274 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-bound-sa-token\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:58.745477 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.745433 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lngx6\" (UniqueName: \"kubernetes.io/projected/4250508f-31ea-4575-b84e-50a57d1aa159-kube-api-access-lngx6\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7\" (UID: \"4250508f-31ea-4575-b84e-50a57d1aa159\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7" Jul 29 20:46:58.746441 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.746066 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxg2d\" (UniqueName: \"kubernetes.io/projected/43c0ba7b-1895-4952-b45b-87f2983002d1-kube-api-access-kxg2d\") pod \"ingress-canary-zwgsv\" (UID: \"43c0ba7b-1895-4952-b45b-87f2983002d1\") " pod="openshift-ingress-canary/ingress-canary-zwgsv" Jul 29 20:46:58.746441 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.746133 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx6r7\" (UniqueName: \"kubernetes.io/projected/863cac4c-638f-4789-88d9-263045a9c8ac-kube-api-access-qx6r7\") pod \"service-ca-operator-5d6c75f84f-vmz8d\" (UID: \"863cac4c-638f-4789-88d9-263045a9c8ac\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vmz8d" Jul 29 20:46:58.746441 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.746369 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkhh5\" (UniqueName: \"kubernetes.io/projected/41604e7f-766a-4d75-853d-371e7f7e6a92-kube-api-access-kkhh5\") pod \"network-check-source-7d88cb55d8-xrtjv\" (UID: \"41604e7f-766a-4d75-853d-371e7f7e6a92\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xrtjv" Jul 29 20:46:58.746441 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.746393 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5b5n\" (UniqueName: \"kubernetes.io/projected/c4622d90-8ccb-4ee8-80a8-6a0a0d4e552d-kube-api-access-p5b5n\") pod \"volume-data-source-validator-549c6d54cf-q9s7m\" (UID: \"c4622d90-8ccb-4ee8-80a8-6a0a0d4e552d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q9s7m" Jul 29 20:46:58.747145 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.747107 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9db9c\" (UniqueName: \"kubernetes.io/projected/ffefc971-2b69-4967-bbd3-35da515c68df-kube-api-access-9db9c\") pod \"cluster-samples-operator-6748489656-nmw5v\" (UID: \"ffefc971-2b69-4967-bbd3-35da515c68df\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v" Jul 29 20:46:58.747637 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.747493 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgf74\" (UniqueName: \"kubernetes.io/projected/1f181cb2-5f41-425b-97d5-6057df483f86-kube-api-access-sgf74\") pod \"cluster-monitoring-operator-548b489cb8-brnnx\" (UID: \"1f181cb2-5f41-425b-97d5-6057df483f86\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" Jul 29 20:46:58.747637 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.747601 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5cws\" (UniqueName: \"kubernetes.io/projected/9749b441-c685-4fa6-8044-791cae15b712-kube-api-access-l5cws\") pod \"dns-default-t89sj\" (UID: \"9749b441-c685-4fa6-8044-791cae15b712\") " pod="openshift-dns/dns-default-t89sj" Jul 29 20:46:58.787861 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.787829 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q9s7m" Jul 29 20:46:58.817810 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.817776 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:46:58.827729 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.827703 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-9hbmn" Jul 29 20:46:58.834460 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.834440 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7" Jul 29 20:46:58.858374 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.858350 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vmz8d" Jul 29 20:46:58.864192 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.864173 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zwgsv" Jul 29 20:46:58.871592 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:58.871573 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xrtjv" Jul 29 20:46:59.240342 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:59.240309 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:46:59.240521 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:59.240348 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-nmw5v\" (UID: \"ffefc971-2b69-4967-bbd3-35da515c68df\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v" Jul 29 20:46:59.240521 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:59.240376 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-brnnx\" (UID: \"1f181cb2-5f41-425b-97d5-6057df483f86\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" Jul 29 20:46:59.240521 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:59.240407 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:46:59.240521 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:59.240458 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-q7z24\" (UID: \"30987044-0cc1-4e94-91d9-4712292608be\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-q7z24" Jul 29 20:46:59.240521 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:59.240487 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:46:59.240521 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:59.240496 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 20:46:59.240521 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:59.240519 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f68896cdc-qpf7g: secret "image-registry-tls" not found Jul 29 20:46:59.240802 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:59.240529 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls\") pod \"dns-default-t89sj\" (UID: \"9749b441-c685-4fa6-8044-791cae15b712\") " pod="openshift-dns/dns-default-t89sj" Jul 29 20:46:59.240802 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:59.240540 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 20:46:59.240802 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:59.240540 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 20:46:59.240802 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:59.240585 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls podName:cc90f71b-6647-4bb2-af21-cfaf42f51393 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:00.240563954 +0000 UTC m=+37.111667404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls") pod "image-registry-7f68896cdc-qpf7g" (UID: "cc90f71b-6647-4bb2-af21-cfaf42f51393") : secret "image-registry-tls" not found Jul 29 20:46:59.240802 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:59.240540 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 20:46:59.240802 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:59.240603 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls podName:ffefc971-2b69-4967-bbd3-35da515c68df nodeName:}" failed. No retries permitted until 2026-07-29 20:47:00.240593643 +0000 UTC m=+37.111697094 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls") pod "cluster-samples-operator-6748489656-nmw5v" (UID: "ffefc971-2b69-4967-bbd3-35da515c68df") : secret "samples-operator-tls" not found Jul 29 20:46:59.240802 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:59.240575 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 20:46:59.240802 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:59.240620 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert podName:30987044-0cc1-4e94-91d9-4712292608be nodeName:}" failed. No retries permitted until 2026-07-29 20:47:00.2406113 +0000 UTC m=+37.111714746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-q7z24" (UID: "30987044-0cc1-4e94-91d9-4712292608be") : secret "networking-console-plugin-cert" not found Jul 29 20:46:59.240802 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:59.240621 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 20:46:59.240802 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:59.240637 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle podName:d56bff06-5480-4bcc-bf5b-dbfd2372f9cd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:00.240628535 +0000 UTC m=+37.111731984 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle") pod "router-default-5f46dc85cf-9562c" (UID: "d56bff06-5480-4bcc-bf5b-dbfd2372f9cd") : configmap references non-existent config key: service-ca.crt Jul 29 20:46:59.240802 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:59.240651 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs podName:d56bff06-5480-4bcc-bf5b-dbfd2372f9cd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:00.240643426 +0000 UTC m=+37.111746876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs") pod "router-default-5f46dc85cf-9562c" (UID: "d56bff06-5480-4bcc-bf5b-dbfd2372f9cd") : secret "router-metrics-certs-default" not found Jul 29 20:46:59.240802 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:59.240668 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls podName:1f181cb2-5f41-425b-97d5-6057df483f86 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:00.240660372 +0000 UTC m=+37.111763830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-brnnx" (UID: "1f181cb2-5f41-425b-97d5-6057df483f86") : secret "cluster-monitoring-operator-tls" not found Jul 29 20:46:59.240802 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:46:59.240683 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls podName:9749b441-c685-4fa6-8044-791cae15b712 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:00.240675602 +0000 UTC m=+37.111779053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls") pod "dns-default-t89sj" (UID: "9749b441-c685-4fa6-8044-791cae15b712") : secret "dns-default-metrics-tls" not found Jul 29 20:46:59.551272 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:59.551176 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:46:59.551424 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:59.551356 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:46:59.553723 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:59.553702 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 20:46:59.553723 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:59.553712 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 20:46:59.553960 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:46:59.553713 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lzlv4\"" Jul 29 20:47:00.061576 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.061476 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7"] Jul 29 20:47:00.066725 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:47:00.066633 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4250508f_31ea_4575_b84e_50a57d1aa159.slice/crio-e17652041d880d3d96202b093ae8050d41f7e25ef800553c66db69a65664a9cf WatchSource:0}: Error finding container e17652041d880d3d96202b093ae8050d41f7e25ef800553c66db69a65664a9cf: Status 404 returned error can't find the container with id e17652041d880d3d96202b093ae8050d41f7e25ef800553c66db69a65664a9cf Jul 29 20:47:00.073574 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.073549 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vmz8d"] Jul 29 20:47:00.074963 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.074897 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q9s7m"] Jul 29 20:47:00.078075 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.078054 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-xrtjv"] Jul 29 20:47:00.081454 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.081292 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zwgsv"] Jul 29 20:47:00.092529 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.092499 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-9hbmn"] Jul 29 20:47:00.095385 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.095362 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-8v2vf"] Jul 29 20:47:00.098446 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:47:00.098423 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08729b6_2792_466e_b7c4_fb0ae44087e6.slice/crio-b1eb2f0d05a6f56720ecebd882d01d3036c0ce9e7ce937636fa490cae182ef9f WatchSource:0}: Error finding container b1eb2f0d05a6f56720ecebd882d01d3036c0ce9e7ce937636fa490cae182ef9f: Status 404 returned error can't find the container with id b1eb2f0d05a6f56720ecebd882d01d3036c0ce9e7ce937636fa490cae182ef9f Jul 29 20:47:00.099956 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:47:00.099734 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c0ba7b_1895_4952_b45b_87f2983002d1.slice/crio-ed4e0fb1d84b0b6edcbacc3ee4ac95948c1204be45c283b781668bfb263f71a3 WatchSource:0}: Error finding container ed4e0fb1d84b0b6edcbacc3ee4ac95948c1204be45c283b781668bfb263f71a3: Status 404 returned error can't find the container with id ed4e0fb1d84b0b6edcbacc3ee4ac95948c1204be45c283b781668bfb263f71a3 Jul 29 20:47:00.251017 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.250980 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:47:00.251187 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.251023 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-nmw5v\" (UID: \"ffefc971-2b69-4967-bbd3-35da515c68df\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v" Jul 29 20:47:00.251187 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.251044 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-brnnx\" (UID: \"1f181cb2-5f41-425b-97d5-6057df483f86\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" Jul 29 20:47:00.251187 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.251074 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:47:00.251187 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.251115 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-q7z24\" (UID: \"30987044-0cc1-4e94-91d9-4712292608be\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-q7z24" Jul 29 20:47:00.251187 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:00.251136 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 20:47:00.251187 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.251141 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:47:00.251187 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:00.251139 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 20:47:00.251436 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:00.251210 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls podName:ffefc971-2b69-4967-bbd3-35da515c68df nodeName:}" failed. No retries permitted until 2026-07-29 20:47:02.251191242 +0000 UTC m=+39.122294722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls") pod "cluster-samples-operator-6748489656-nmw5v" (UID: "ffefc971-2b69-4967-bbd3-35da515c68df") : secret "samples-operator-tls" not found Jul 29 20:47:00.251436 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:00.251220 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f68896cdc-qpf7g: secret "image-registry-tls" not found Jul 29 20:47:00.251436 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:00.251193 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 20:47:00.251436 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:00.251261 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle podName:d56bff06-5480-4bcc-bf5b-dbfd2372f9cd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:02.251243419 +0000 UTC m=+39.122346869 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle") pod "router-default-5f46dc85cf-9562c" (UID: "d56bff06-5480-4bcc-bf5b-dbfd2372f9cd") : configmap references non-existent config key: service-ca.crt Jul 29 20:47:00.251436 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:00.251209 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 20:47:00.251436 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:00.251211 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 20:47:00.251436 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:00.251292 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls podName:cc90f71b-6647-4bb2-af21-cfaf42f51393 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:02.251278586 +0000 UTC m=+39.122382032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls") pod "image-registry-7f68896cdc-qpf7g" (UID: "cc90f71b-6647-4bb2-af21-cfaf42f51393") : secret "image-registry-tls" not found Jul 29 20:47:00.251436 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:00.251317 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert podName:30987044-0cc1-4e94-91d9-4712292608be nodeName:}" failed. No retries permitted until 2026-07-29 20:47:02.25130797 +0000 UTC m=+39.122411422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-q7z24" (UID: "30987044-0cc1-4e94-91d9-4712292608be") : secret "networking-console-plugin-cert" not found Jul 29 20:47:00.251436 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:00.251329 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls podName:1f181cb2-5f41-425b-97d5-6057df483f86 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:02.251324413 +0000 UTC m=+39.122427858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-brnnx" (UID: "1f181cb2-5f41-425b-97d5-6057df483f86") : secret "cluster-monitoring-operator-tls" not found Jul 29 20:47:00.251436 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:00.251341 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs podName:d56bff06-5480-4bcc-bf5b-dbfd2372f9cd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:02.25133687 +0000 UTC m=+39.122440314 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs") pod "router-default-5f46dc85cf-9562c" (UID: "d56bff06-5480-4bcc-bf5b-dbfd2372f9cd") : secret "router-metrics-certs-default" not found Jul 29 20:47:00.251436 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.251364 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls\") pod \"dns-default-t89sj\" (UID: \"9749b441-c685-4fa6-8044-791cae15b712\") " pod="openshift-dns/dns-default-t89sj" Jul 29 20:47:00.251749 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:00.251448 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 20:47:00.251749 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:00.251485 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls podName:9749b441-c685-4fa6-8044-791cae15b712 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:02.251475158 +0000 UTC m=+39.122578605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls") pod "dns-default-t89sj" (UID: "9749b441-c685-4fa6-8044-791cae15b712") : secret "dns-default-metrics-tls" not found Jul 29 20:47:00.720847 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.720618 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" event={"ID":"bf9e0545-985e-47df-af42-4453c1e24937","Type":"ContainerStarted","Data":"a38c9e95561fd5c408f8718bcb84c15895d1ef4cf5bd2959d02ba87b654309e8"} Jul 29 20:47:00.722289 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.722236 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7" event={"ID":"4250508f-31ea-4575-b84e-50a57d1aa159","Type":"ContainerStarted","Data":"e17652041d880d3d96202b093ae8050d41f7e25ef800553c66db69a65664a9cf"} Jul 29 20:47:00.723476 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.723452 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zwgsv" event={"ID":"43c0ba7b-1895-4952-b45b-87f2983002d1","Type":"ContainerStarted","Data":"ed4e0fb1d84b0b6edcbacc3ee4ac95948c1204be45c283b781668bfb263f71a3"} Jul 29 20:47:00.724827 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.724775 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-9hbmn" event={"ID":"a08729b6-2792-466e-b7c4-fb0ae44087e6","Type":"ContainerStarted","Data":"b1eb2f0d05a6f56720ecebd882d01d3036c0ce9e7ce937636fa490cae182ef9f"} Jul 29 20:47:00.729282 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.729160 2562 generic.go:356] "Generic (PLEG): container finished" podID="a11939fc-63a4-4f61-9de5-5d9bef849715" containerID="9839e3cd645548452185d5a26ed0a4100e66267bef3454072552bf790d05a12e" exitCode=0 Jul 29 20:47:00.729282 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.729240 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9nwx" event={"ID":"a11939fc-63a4-4f61-9de5-5d9bef849715","Type":"ContainerDied","Data":"9839e3cd645548452185d5a26ed0a4100e66267bef3454072552bf790d05a12e"} Jul 29 20:47:00.731875 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.730663 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xrtjv" event={"ID":"41604e7f-766a-4d75-853d-371e7f7e6a92","Type":"ContainerStarted","Data":"6462bc860266541e4e70b53975c44580844ac6700920c5d6a4996227ad614d85"} Jul 29 20:47:00.732149 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.732116 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vmz8d" event={"ID":"863cac4c-638f-4789-88d9-263045a9c8ac","Type":"ContainerStarted","Data":"30473d44415d9fe7a03f85f2808fe1e37c153955b999d4a6ee931471d87aeedd"} Jul 29 20:47:00.734097 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:00.734064 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q9s7m" event={"ID":"c4622d90-8ccb-4ee8-80a8-6a0a0d4e552d","Type":"ContainerStarted","Data":"f55eaf54cf71cc8f5a477228793ede3ea531ada57e2609d2b77357bbfc710109"} Jul 29 20:47:01.745187 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:01.744932 2562 generic.go:356] "Generic (PLEG): container finished" podID="a11939fc-63a4-4f61-9de5-5d9bef849715" containerID="086f45ee7c389640415b19cfd0acf8c294ba95081aa983329ae639c7a23a03ee" exitCode=0 Jul 29 20:47:01.745187 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:01.744988 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9nwx" event={"ID":"a11939fc-63a4-4f61-9de5-5d9bef849715","Type":"ContainerDied","Data":"086f45ee7c389640415b19cfd0acf8c294ba95081aa983329ae639c7a23a03ee"} Jul 29 20:47:02.271607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:02.270399 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:47:02.271607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:02.270457 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-nmw5v\" (UID: \"ffefc971-2b69-4967-bbd3-35da515c68df\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v" Jul 29 20:47:02.271607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:02.270487 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-brnnx\" (UID: \"1f181cb2-5f41-425b-97d5-6057df483f86\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" Jul 29 20:47:02.271607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:02.270519 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:47:02.271607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:02.270611 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-q7z24\" (UID: \"30987044-0cc1-4e94-91d9-4712292608be\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-q7z24" Jul 29 20:47:02.271607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:02.270639 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:47:02.271607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:02.270746 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls\") pod \"dns-default-t89sj\" (UID: \"9749b441-c685-4fa6-8044-791cae15b712\") " pod="openshift-dns/dns-default-t89sj" Jul 29 20:47:02.271607 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:02.270992 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 20:47:02.271607 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:02.271103 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls podName:9749b441-c685-4fa6-8044-791cae15b712 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:06.271083312 +0000 UTC m=+43.142186760 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls") pod "dns-default-t89sj" (UID: "9749b441-c685-4fa6-8044-791cae15b712") : secret "dns-default-metrics-tls" not found Jul 29 20:47:02.271607 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:02.271134 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 20:47:02.271607 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:02.271206 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls podName:1f181cb2-5f41-425b-97d5-6057df483f86 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:06.271185579 +0000 UTC m=+43.142289045 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-brnnx" (UID: "1f181cb2-5f41-425b-97d5-6057df483f86") : secret "cluster-monitoring-operator-tls" not found Jul 29 20:47:02.271607 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:02.271284 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 20:47:02.271607 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:02.271296 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f68896cdc-qpf7g: secret "image-registry-tls" not found Jul 29 20:47:02.271607 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:02.271334 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls podName:cc90f71b-6647-4bb2-af21-cfaf42f51393 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:06.271321455 +0000 UTC m=+43.142424903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls") pod "image-registry-7f68896cdc-qpf7g" (UID: "cc90f71b-6647-4bb2-af21-cfaf42f51393") : secret "image-registry-tls" not found Jul 29 20:47:02.271607 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:02.271392 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 20:47:02.271607 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:02.271430 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls podName:ffefc971-2b69-4967-bbd3-35da515c68df nodeName:}" failed. No retries permitted until 2026-07-29 20:47:06.271417784 +0000 UTC m=+43.142521228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls") pod "cluster-samples-operator-6748489656-nmw5v" (UID: "ffefc971-2b69-4967-bbd3-35da515c68df") : secret "samples-operator-tls" not found Jul 29 20:47:02.272673 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:02.271489 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 20:47:02.272673 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:02.271523 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert podName:30987044-0cc1-4e94-91d9-4712292608be nodeName:}" failed. No retries permitted until 2026-07-29 20:47:06.271511724 +0000 UTC m=+43.142615172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-q7z24" (UID: "30987044-0cc1-4e94-91d9-4712292608be") : secret "networking-console-plugin-cert" not found Jul 29 20:47:02.272673 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:02.271568 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle podName:d56bff06-5480-4bcc-bf5b-dbfd2372f9cd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:06.271553735 +0000 UTC m=+43.142657186 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle") pod "router-default-5f46dc85cf-9562c" (UID: "d56bff06-5480-4bcc-bf5b-dbfd2372f9cd") : configmap references non-existent config key: service-ca.crt Jul 29 20:47:02.272673 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:02.271606 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 20:47:02.272673 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:02.271657 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs podName:d56bff06-5480-4bcc-bf5b-dbfd2372f9cd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:06.271641754 +0000 UTC m=+43.142745215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs") pod "router-default-5f46dc85cf-9562c" (UID: "d56bff06-5480-4bcc-bf5b-dbfd2372f9cd") : secret "router-metrics-certs-default" not found Jul 29 20:47:02.765237 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:02.764699 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9nwx" event={"ID":"a11939fc-63a4-4f61-9de5-5d9bef849715","Type":"ContainerStarted","Data":"007bdd931a070b7e1680fa94ccad99a825ede22a09c352f3351b341191c5b3ed"} Jul 29 20:47:02.786453 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:02.786388 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v9nwx" podStartSLOduration=10.285261581 podStartE2EDuration="39.786365604s" totalImagesPullingTime="29.501104023s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:02.783868771 +0000 UTC m=+39.654972238" watchObservedRunningTime="2026-07-29 20:47:02.786365604 +0000 UTC m=+39.657469072" Jul 29 20:47:06.309528 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:06.309486 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:47:06.310117 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:06.309536 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-nmw5v\" (UID: \"ffefc971-2b69-4967-bbd3-35da515c68df\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v" Jul 29 20:47:06.310117 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:06.309564 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-brnnx\" (UID: \"1f181cb2-5f41-425b-97d5-6057df483f86\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" Jul 29 20:47:06.310117 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:06.309598 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:47:06.310117 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:06.309637 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 20:47:06.310117 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:06.309661 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f68896cdc-qpf7g: secret "image-registry-tls" not found Jul 29 20:47:06.310117 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:06.309677 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 20:47:06.310117 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:06.309697 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 20:47:06.310117 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:06.309716 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls podName:cc90f71b-6647-4bb2-af21-cfaf42f51393 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:14.309700333 +0000 UTC m=+51.180803777 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls") pod "image-registry-7f68896cdc-qpf7g" (UID: "cc90f71b-6647-4bb2-af21-cfaf42f51393") : secret "image-registry-tls" not found Jul 29 20:47:06.310117 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:06.309735 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs podName:d56bff06-5480-4bcc-bf5b-dbfd2372f9cd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:14.30972578 +0000 UTC m=+51.180829225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs") pod "router-default-5f46dc85cf-9562c" (UID: "d56bff06-5480-4bcc-bf5b-dbfd2372f9cd") : secret "router-metrics-certs-default" not found Jul 29 20:47:06.310117 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:06.309750 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls podName:ffefc971-2b69-4967-bbd3-35da515c68df nodeName:}" failed. No retries permitted until 2026-07-29 20:47:14.309742252 +0000 UTC m=+51.180845700 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls") pod "cluster-samples-operator-6748489656-nmw5v" (UID: "ffefc971-2b69-4967-bbd3-35da515c68df") : secret "samples-operator-tls" not found Jul 29 20:47:06.310117 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:06.309754 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 20:47:06.310117 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:06.309791 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls podName:1f181cb2-5f41-425b-97d5-6057df483f86 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:14.309775621 +0000 UTC m=+51.180879067 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-brnnx" (UID: "1f181cb2-5f41-425b-97d5-6057df483f86") : secret "cluster-monitoring-operator-tls" not found Jul 29 20:47:06.310117 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:06.309817 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-q7z24\" (UID: \"30987044-0cc1-4e94-91d9-4712292608be\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-q7z24" Jul 29 20:47:06.310117 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:06.309836 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:47:06.310117 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:06.309876 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls\") pod \"dns-default-t89sj\" (UID: \"9749b441-c685-4fa6-8044-791cae15b712\") " pod="openshift-dns/dns-default-t89sj" Jul 29 20:47:06.310117 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:06.309897 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 20:47:06.310695 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:06.309946 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert podName:30987044-0cc1-4e94-91d9-4712292608be nodeName:}" failed. No retries permitted until 2026-07-29 20:47:14.309935571 +0000 UTC m=+51.181039019 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-q7z24" (UID: "30987044-0cc1-4e94-91d9-4712292608be") : secret "networking-console-plugin-cert" not found Jul 29 20:47:06.310695 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:06.309967 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 20:47:06.310695 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:06.309991 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls podName:9749b441-c685-4fa6-8044-791cae15b712 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:14.309983136 +0000 UTC m=+51.181086581 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls") pod "dns-default-t89sj" (UID: "9749b441-c685-4fa6-8044-791cae15b712") : secret "dns-default-metrics-tls" not found Jul 29 20:47:06.310695 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:06.310011 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle podName:d56bff06-5480-4bcc-bf5b-dbfd2372f9cd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:14.310001262 +0000 UTC m=+51.181104734 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle") pod "router-default-5f46dc85cf-9562c" (UID: "d56bff06-5480-4bcc-bf5b-dbfd2372f9cd") : configmap references non-existent config key: service-ca.crt Jul 29 20:47:08.691544 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.691499 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vgs2s"] Jul 29 20:47:08.726173 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.726134 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vgs2s"] Jul 29 20:47:08.726340 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.726292 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:08.728156 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.728135 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2kmm5\"" Jul 29 20:47:08.728465 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.728178 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 20:47:08.728716 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.728217 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 20:47:08.783425 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.783393 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/0.log" Jul 29 20:47:08.783606 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.783440 2562 generic.go:356] "Generic (PLEG): container finished" podID="bf9e0545-985e-47df-af42-4453c1e24937" containerID="9e63a6f992d088500ff12ea6f0f189ebce95e3fbb880b38b30cd7377bc71218c" exitCode=255 Jul 29 20:47:08.783606 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.783525 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" event={"ID":"bf9e0545-985e-47df-af42-4453c1e24937","Type":"ContainerDied","Data":"9e63a6f992d088500ff12ea6f0f189ebce95e3fbb880b38b30cd7377bc71218c"} Jul 29 20:47:08.783970 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.783950 2562 scope.go:117] "RemoveContainer" containerID="9e63a6f992d088500ff12ea6f0f189ebce95e3fbb880b38b30cd7377bc71218c" Jul 29 20:47:08.785606 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.785580 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7" event={"ID":"4250508f-31ea-4575-b84e-50a57d1aa159","Type":"ContainerStarted","Data":"2b8a78557610d2d3c452c2089e07939d73d54f14fef2dffd52a7226e6f09cab7"} Jul 29 20:47:08.789101 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.789046 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zwgsv" event={"ID":"43c0ba7b-1895-4952-b45b-87f2983002d1","Type":"ContainerStarted","Data":"8781347b2de8dab9bf515ee1b401ba881df297506459bb389c35ccdb01a65f64"} Jul 29 20:47:08.791020 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.790991 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-9hbmn" event={"ID":"a08729b6-2792-466e-b7c4-fb0ae44087e6","Type":"ContainerStarted","Data":"16d3e61abc4a0defa0096fd045f572c80a8d3e112c00aae109ca52e5d53ed53e"} Jul 29 20:47:08.792532 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.792508 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xrtjv" event={"ID":"41604e7f-766a-4d75-853d-371e7f7e6a92","Type":"ContainerStarted","Data":"59664b1f8c19e4ecb98406d0422e43d8fb097dce021dbf9a18537cb1d5bfdf6e"} Jul 29 20:47:08.794066 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.794039 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vmz8d" event={"ID":"863cac4c-638f-4789-88d9-263045a9c8ac","Type":"ContainerStarted","Data":"c3362fc539261f9fa82f21590c69ea1956e9d4c70a60be5b4ec26b89cb88cf10"} Jul 29 20:47:08.795709 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.795689 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q9s7m" event={"ID":"c4622d90-8ccb-4ee8-80a8-6a0a0d4e552d","Type":"ContainerStarted","Data":"5c7a63e979ec383dcfdbacbe65b667909063e590cfd288e2a7428fae1e2a328c"} Jul 29 20:47:08.816818 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.816765 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xrtjv" podStartSLOduration=31.966720586 podStartE2EDuration="39.816747506s" totalImagesPullingTime="7.85002692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:08.815553154 +0000 UTC m=+45.686656624" watchObservedRunningTime="2026-07-29 20:47:08.816747506 +0000 UTC m=+45.687850976" Jul 29 20:47:08.818112 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.818094 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:47:08.818201 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.818132 2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:47:08.831330 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.831281 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q9s7m" podStartSLOduration=33.297997929 podStartE2EDuration="39.831263835s" totalImagesPullingTime="6.533265906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:08.830100756 +0000 UTC m=+45.701204225" watchObservedRunningTime="2026-07-29 20:47:08.831263835 +0000 UTC m=+45.702367305" Jul 29 20:47:08.833034 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.833013 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chhfn\" (UniqueName: \"kubernetes.io/projected/053c21f8-14cd-4244-a80d-9713ee979ca3-kube-api-access-chhfn\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:08.833193 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.833171 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/053c21f8-14cd-4244-a80d-9713ee979ca3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:08.833313 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.833240 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/053c21f8-14cd-4244-a80d-9713ee979ca3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:08.833838 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.833744 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/053c21f8-14cd-4244-a80d-9713ee979ca3-crio-socket\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:08.833838 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.833808 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/053c21f8-14cd-4244-a80d-9713ee979ca3-data-volume\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:08.846856 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.846814 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-9hbmn" podStartSLOduration=31.983089805 podStartE2EDuration="39.846798782s" totalImagesPullingTime="7.863708977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:08.845970603 +0000 UTC m=+45.717074073" watchObservedRunningTime="2026-07-29 20:47:08.846798782 +0000 UTC m=+45.717902469" Jul 29 20:47:08.862393 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.861312 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7" podStartSLOduration=31.970081134 podStartE2EDuration="39.861293888s" totalImagesPullingTime="7.891212754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:08.860321605 +0000 UTC m=+45.731425073" watchObservedRunningTime="2026-07-29 20:47:08.861293888 +0000 UTC m=+45.732397356" Jul 29 20:47:08.884279 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.884240 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vmz8d" podStartSLOduration=32.031900603 podStartE2EDuration="39.884223562s" totalImagesPullingTime="7.852322959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:08.883075379 +0000 UTC m=+45.754178859" watchObservedRunningTime="2026-07-29 20:47:08.884223562 +0000 UTC m=+45.755327028" Jul 29 20:47:08.907077 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.907014 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zwgsv" podStartSLOduration=3.053795543 podStartE2EDuration="10.906994265s" totalImagesPullingTime="7.853198722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:08.905701342 +0000 UTC m=+45.776804810" watchObservedRunningTime="2026-07-29 20:47:08.906994265 +0000 UTC m=+45.778097744" Jul 29 20:47:08.935938 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.935875 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/053c21f8-14cd-4244-a80d-9713ee979ca3-crio-socket\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:08.936125 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.935953 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/053c21f8-14cd-4244-a80d-9713ee979ca3-data-volume\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:08.936207 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.936128 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chhfn\" (UniqueName: \"kubernetes.io/projected/053c21f8-14cd-4244-a80d-9713ee979ca3-kube-api-access-chhfn\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:08.936207 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.936188 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/053c21f8-14cd-4244-a80d-9713ee979ca3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:08.936325 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.936220 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/053c21f8-14cd-4244-a80d-9713ee979ca3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:08.936378 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:08.936358 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 20:47:08.936435 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:08.936427 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/053c21f8-14cd-4244-a80d-9713ee979ca3-insights-runtime-extractor-tls podName:053c21f8-14cd-4244-a80d-9713ee979ca3 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:09.43640793 +0000 UTC m=+46.307511379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/053c21f8-14cd-4244-a80d-9713ee979ca3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vgs2s" (UID: "053c21f8-14cd-4244-a80d-9713ee979ca3") : secret "insights-runtime-extractor-tls" not found Jul 29 20:47:08.936820 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.936791 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/053c21f8-14cd-4244-a80d-9713ee979ca3-crio-socket\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:08.937176 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.937153 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/053c21f8-14cd-4244-a80d-9713ee979ca3-data-volume\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:08.937480 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:08.937460 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 20:47:08.937569 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:08.937519 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/053c21f8-14cd-4244-a80d-9713ee979ca3-kube-rbac-proxy-cm podName:053c21f8-14cd-4244-a80d-9713ee979ca3 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:09.437500958 +0000 UTC m=+46.308604402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/053c21f8-14cd-4244-a80d-9713ee979ca3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vgs2s" (UID: "053c21f8-14cd-4244-a80d-9713ee979ca3") : configmap "kube-rbac-proxy" not found Jul 29 20:47:08.947455 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:08.947298 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chhfn\" (UniqueName: \"kubernetes.io/projected/053c21f8-14cd-4244-a80d-9713ee979ca3-kube-api-access-chhfn\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:09.441070 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:09.441026 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/053c21f8-14cd-4244-a80d-9713ee979ca3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:09.441270 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:09.441084 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/053c21f8-14cd-4244-a80d-9713ee979ca3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:09.441270 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:09.441154 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 20:47:09.441270 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:09.441235 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/053c21f8-14cd-4244-a80d-9713ee979ca3-kube-rbac-proxy-cm podName:053c21f8-14cd-4244-a80d-9713ee979ca3 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:10.441214688 +0000 UTC m=+47.312318154 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/053c21f8-14cd-4244-a80d-9713ee979ca3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vgs2s" (UID: "053c21f8-14cd-4244-a80d-9713ee979ca3") : configmap "kube-rbac-proxy" not found Jul 29 20:47:09.441270 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:09.441246 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 20:47:09.441482 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:09.441299 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/053c21f8-14cd-4244-a80d-9713ee979ca3-insights-runtime-extractor-tls podName:053c21f8-14cd-4244-a80d-9713ee979ca3 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:10.441285225 +0000 UTC m=+47.312388689 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/053c21f8-14cd-4244-a80d-9713ee979ca3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vgs2s" (UID: "053c21f8-14cd-4244-a80d-9713ee979ca3") : secret "insights-runtime-extractor-tls" not found Jul 29 20:47:09.803748 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:09.803666 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 20:47:09.804253 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:09.804214 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/0.log" Jul 29 20:47:09.804310 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:09.804255 2562 generic.go:356] "Generic (PLEG): container finished" podID="bf9e0545-985e-47df-af42-4453c1e24937" containerID="7664a960dee6ea88886d41ea03b6daa612cae9ce2dc520dfafa93aea302e3351" exitCode=255 Jul 29 20:47:09.804363 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:09.804331 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" event={"ID":"bf9e0545-985e-47df-af42-4453c1e24937","Type":"ContainerDied","Data":"7664a960dee6ea88886d41ea03b6daa612cae9ce2dc520dfafa93aea302e3351"} Jul 29 20:47:09.804427 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:09.804412 2562 scope.go:117] "RemoveContainer" containerID="9e63a6f992d088500ff12ea6f0f189ebce95e3fbb880b38b30cd7377bc71218c" Jul 29 20:47:09.804671 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:09.804649 2562 scope.go:117] "RemoveContainer" containerID="7664a960dee6ea88886d41ea03b6daa612cae9ce2dc520dfafa93aea302e3351" Jul 29 20:47:09.804872 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:09.804844 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-8v2vf_openshift-console-operator(bf9e0545-985e-47df-af42-4453c1e24937)\"" pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" podUID="bf9e0545-985e-47df-af42-4453c1e24937" Jul 29 20:47:10.451304 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:10.451266 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/053c21f8-14cd-4244-a80d-9713ee979ca3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:10.451511 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:10.451312 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/053c21f8-14cd-4244-a80d-9713ee979ca3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:10.451511 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:10.451390 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 20:47:10.451511 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:10.451476 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 20:47:10.451637 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:10.451480 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/053c21f8-14cd-4244-a80d-9713ee979ca3-kube-rbac-proxy-cm podName:053c21f8-14cd-4244-a80d-9713ee979ca3 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:12.451462428 +0000 UTC m=+49.322565876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/053c21f8-14cd-4244-a80d-9713ee979ca3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vgs2s" (UID: "053c21f8-14cd-4244-a80d-9713ee979ca3") : configmap "kube-rbac-proxy" not found Jul 29 20:47:10.451637 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:10.451541 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/053c21f8-14cd-4244-a80d-9713ee979ca3-insights-runtime-extractor-tls podName:053c21f8-14cd-4244-a80d-9713ee979ca3 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:12.45152369 +0000 UTC m=+49.322627142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/053c21f8-14cd-4244-a80d-9713ee979ca3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vgs2s" (UID: "053c21f8-14cd-4244-a80d-9713ee979ca3") : secret "insights-runtime-extractor-tls" not found Jul 29 20:47:10.703591 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:10.703513 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l4txq_43dd2f3d-02e2-47ba-b20a-4365c2c6f45b/dns-node-resolver/0.log" Jul 29 20:47:10.808998 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:10.808966 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 20:47:10.809366 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:10.809312 2562 scope.go:117] "RemoveContainer" containerID="7664a960dee6ea88886d41ea03b6daa612cae9ce2dc520dfafa93aea302e3351" Jul 29 20:47:10.809491 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:10.809469 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-8v2vf_openshift-console-operator(bf9e0545-985e-47df-af42-4453c1e24937)\"" pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" podUID="bf9e0545-985e-47df-af42-4453c1e24937" Jul 29 20:47:11.906537 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:11.906512 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p87w8_e6d19b51-bb38-465c-ae75-cfa9d195bb72/node-ca/0.log" Jul 29 20:47:12.471297 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:12.471254 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/053c21f8-14cd-4244-a80d-9713ee979ca3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:12.471297 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:12.471299 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/053c21f8-14cd-4244-a80d-9713ee979ca3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:12.471516 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:12.471376 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 20:47:12.471516 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:12.471394 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 20:47:12.471516 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:12.471447 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/053c21f8-14cd-4244-a80d-9713ee979ca3-insights-runtime-extractor-tls podName:053c21f8-14cd-4244-a80d-9713ee979ca3 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:16.471432841 +0000 UTC m=+53.342536286 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/053c21f8-14cd-4244-a80d-9713ee979ca3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vgs2s" (UID: "053c21f8-14cd-4244-a80d-9713ee979ca3") : secret "insights-runtime-extractor-tls" not found Jul 29 20:47:12.471516 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:12.471463 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/053c21f8-14cd-4244-a80d-9713ee979ca3-kube-rbac-proxy-cm podName:053c21f8-14cd-4244-a80d-9713ee979ca3 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:16.471455438 +0000 UTC m=+53.342558883 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/053c21f8-14cd-4244-a80d-9713ee979ca3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vgs2s" (UID: "053c21f8-14cd-4244-a80d-9713ee979ca3") : configmap "kube-rbac-proxy" not found Jul 29 20:47:12.705557 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:12.705523 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zwgsv_43c0ba7b-1895-4952-b45b-87f2983002d1/serve-healthcheck-canary/0.log" Jul 29 20:47:14.389683 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:14.389634 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:47:14.390095 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:14.389692 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-nmw5v\" (UID: \"ffefc971-2b69-4967-bbd3-35da515c68df\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v" Jul 29 20:47:14.390095 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:14.389726 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-brnnx\" (UID: \"1f181cb2-5f41-425b-97d5-6057df483f86\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" Jul 29 20:47:14.390095 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:14.389760 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:47:14.390095 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:14.389798 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 20:47:14.390095 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:14.389818 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f68896cdc-qpf7g: secret "image-registry-tls" not found Jul 29 20:47:14.390095 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:14.389848 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 20:47:14.390095 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:14.389865 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 20:47:14.390095 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:14.389893 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls podName:cc90f71b-6647-4bb2-af21-cfaf42f51393 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:30.389873873 +0000 UTC m=+67.260977376 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls") pod "image-registry-7f68896cdc-qpf7g" (UID: "cc90f71b-6647-4bb2-af21-cfaf42f51393") : secret "image-registry-tls" not found Jul 29 20:47:14.390095 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:14.389944 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls podName:ffefc971-2b69-4967-bbd3-35da515c68df nodeName:}" failed. No retries permitted until 2026-07-29 20:47:30.389913047 +0000 UTC m=+67.261016491 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls") pod "cluster-samples-operator-6748489656-nmw5v" (UID: "ffefc971-2b69-4967-bbd3-35da515c68df") : secret "samples-operator-tls" not found Jul 29 20:47:14.390095 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:14.389894 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 20:47:14.390095 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:14.389968 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs podName:d56bff06-5480-4bcc-bf5b-dbfd2372f9cd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:30.389958708 +0000 UTC m=+67.261062155 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs") pod "router-default-5f46dc85cf-9562c" (UID: "d56bff06-5480-4bcc-bf5b-dbfd2372f9cd") : secret "router-metrics-certs-default" not found Jul 29 20:47:14.390095 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:14.390011 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls podName:1f181cb2-5f41-425b-97d5-6057df483f86 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:30.389996965 +0000 UTC m=+67.261100410 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-brnnx" (UID: "1f181cb2-5f41-425b-97d5-6057df483f86") : secret "cluster-monitoring-operator-tls" not found Jul 29 20:47:14.390095 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:14.390032 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-q7z24\" (UID: \"30987044-0cc1-4e94-91d9-4712292608be\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-q7z24" Jul 29 20:47:14.390095 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:14.390060 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:47:14.390546 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:14.390112 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 20:47:14.390546 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:14.390132 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle podName:d56bff06-5480-4bcc-bf5b-dbfd2372f9cd nodeName:}" failed. No retries permitted until 2026-07-29 20:47:30.390118588 +0000 UTC m=+67.261222039 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle") pod "router-default-5f46dc85cf-9562c" (UID: "d56bff06-5480-4bcc-bf5b-dbfd2372f9cd") : configmap references non-existent config key: service-ca.crt Jul 29 20:47:14.390546 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:14.390111 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls\") pod \"dns-default-t89sj\" (UID: \"9749b441-c685-4fa6-8044-791cae15b712\") " pod="openshift-dns/dns-default-t89sj" Jul 29 20:47:14.390546 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:14.390151 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert podName:30987044-0cc1-4e94-91d9-4712292608be nodeName:}" failed. No retries permitted until 2026-07-29 20:47:30.390142906 +0000 UTC m=+67.261246351 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-q7z24" (UID: "30987044-0cc1-4e94-91d9-4712292608be") : secret "networking-console-plugin-cert" not found Jul 29 20:47:14.390546 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:14.390158 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 20:47:14.390546 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:14.390181 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls podName:9749b441-c685-4fa6-8044-791cae15b712 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:30.390174228 +0000 UTC m=+67.261277673 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls") pod "dns-default-t89sj" (UID: "9749b441-c685-4fa6-8044-791cae15b712") : secret "dns-default-metrics-tls" not found Jul 29 20:47:16.510863 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:16.510824 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/053c21f8-14cd-4244-a80d-9713ee979ca3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:16.510863 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:16.510862 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/053c21f8-14cd-4244-a80d-9713ee979ca3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:16.511319 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:16.511058 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 20:47:16.511319 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:16.511127 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/053c21f8-14cd-4244-a80d-9713ee979ca3-insights-runtime-extractor-tls podName:053c21f8-14cd-4244-a80d-9713ee979ca3 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:24.511110516 +0000 UTC m=+61.382213961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/053c21f8-14cd-4244-a80d-9713ee979ca3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vgs2s" (UID: "053c21f8-14cd-4244-a80d-9713ee979ca3") : secret "insights-runtime-extractor-tls" not found Jul 29 20:47:16.511482 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:16.511462 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/053c21f8-14cd-4244-a80d-9713ee979ca3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:18.818458 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:18.818405 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:47:18.819023 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:18.818505 2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:47:18.819023 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:18.818821 2562 scope.go:117] "RemoveContainer" containerID="7664a960dee6ea88886d41ea03b6daa612cae9ce2dc520dfafa93aea302e3351" Jul 29 20:47:18.819023 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:18.818995 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-8v2vf_openshift-console-operator(bf9e0545-985e-47df-af42-4453c1e24937)\"" pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" podUID="bf9e0545-985e-47df-af42-4453c1e24937" Jul 29 20:47:23.713794 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:23.713761 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tplf2" Jul 29 20:47:24.583634 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:24.583600 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/053c21f8-14cd-4244-a80d-9713ee979ca3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:24.585842 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:24.585819 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/053c21f8-14cd-4244-a80d-9713ee979ca3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vgs2s\" (UID: \"053c21f8-14cd-4244-a80d-9713ee979ca3\") " pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:24.639075 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:24.639045 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2kmm5\"" Jul 29 20:47:24.647555 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:24.647534 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vgs2s" Jul 29 20:47:24.800316 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:24.800289 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vgs2s"] Jul 29 20:47:24.802792 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:47:24.802766 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053c21f8_14cd_4244_a80d_9713ee979ca3.slice/crio-3b688c44c75c05cc328dc4ad75ce07708ab0f0dff078f17b1ba0ae49d50f3fb7 WatchSource:0}: Error finding container 3b688c44c75c05cc328dc4ad75ce07708ab0f0dff078f17b1ba0ae49d50f3fb7: Status 404 returned error can't find the container with id 3b688c44c75c05cc328dc4ad75ce07708ab0f0dff078f17b1ba0ae49d50f3fb7 Jul 29 20:47:24.851522 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:24.851465 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vgs2s" event={"ID":"053c21f8-14cd-4244-a80d-9713ee979ca3","Type":"ContainerStarted","Data":"3b688c44c75c05cc328dc4ad75ce07708ab0f0dff078f17b1ba0ae49d50f3fb7"} Jul 29 20:47:29.226822 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:29.226772 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs\") pod \"network-metrics-daemon-5sgzv\" (UID: \"347e06fd-9f35-47dd-9442-ddd628e5bd35\") " pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:47:29.227359 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:29.226999 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret\") pod \"global-pull-secret-syncer-fhgt5\" (UID: \"ba4aa585-620a-4c88-8c46-09b343186c70\") " pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:47:29.229350 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:29.229328 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 20:47:29.229657 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:29.229637 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 20:47:29.240328 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:29.240306 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347e06fd-9f35-47dd-9442-ddd628e5bd35-metrics-certs\") pod \"network-metrics-daemon-5sgzv\" (UID: \"347e06fd-9f35-47dd-9442-ddd628e5bd35\") " pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:47:29.240442 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:29.240312 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba4aa585-620a-4c88-8c46-09b343186c70-original-pull-secret\") pod \"global-pull-secret-syncer-fhgt5\" (UID: \"ba4aa585-620a-4c88-8c46-09b343186c70\") " pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:47:29.263998 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:29.263969 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhgt5" Jul 29 20:47:29.270118 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:29.270101 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lzlv4\"" Jul 29 20:47:29.278596 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:29.278573 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sgzv" Jul 29 20:47:29.327614 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:29.327578 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m98sf\" (UniqueName: \"kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf\") pod \"network-check-target-7jd4g\" (UID: \"4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5\") " pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:47:29.330394 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:29.330363 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m98sf\" (UniqueName: \"kubernetes.io/projected/4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5-kube-api-access-m98sf\") pod \"network-check-target-7jd4g\" (UID: \"4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5\") " pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:47:29.487421 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:29.487338 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zcmz5\"" Jul 29 20:47:29.495860 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:29.495827 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:47:29.765509 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:29.765476 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fhgt5"] Jul 29 20:47:29.771032 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:47:29.770993 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4aa585_620a_4c88_8c46_09b343186c70.slice/crio-16e45e1218e2b7ef8640259b64f52b8c33144523dbb4dd712e06971002afefa2 WatchSource:0}: Error finding container 16e45e1218e2b7ef8640259b64f52b8c33144523dbb4dd712e06971002afefa2: Status 404 returned error can't find the container with id 16e45e1218e2b7ef8640259b64f52b8c33144523dbb4dd712e06971002afefa2 Jul 29 20:47:29.868610 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:29.868573 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vgs2s" event={"ID":"053c21f8-14cd-4244-a80d-9713ee979ca3","Type":"ContainerStarted","Data":"42b7387da976f13696f44121512f7d383102e8f4fc92190bdfb6209622d44ad9"} Jul 29 20:47:29.869561 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:29.869532 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fhgt5" event={"ID":"ba4aa585-620a-4c88-8c46-09b343186c70","Type":"ContainerStarted","Data":"16e45e1218e2b7ef8640259b64f52b8c33144523dbb4dd712e06971002afefa2"} Jul 29 20:47:29.992844 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:29.992820 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5sgzv"] Jul 29 20:47:29.995119 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:29.995081 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7jd4g"] Jul 29 20:47:29.995237 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:47:29.995075 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347e06fd_9f35_47dd_9442_ddd628e5bd35.slice/crio-75a9b1aa09b2d682eb5745084cd49704ac1ac36dec59ea8fc05e1a925135c67e WatchSource:0}: Error finding container 75a9b1aa09b2d682eb5745084cd49704ac1ac36dec59ea8fc05e1a925135c67e: Status 404 returned error can't find the container with id 75a9b1aa09b2d682eb5745084cd49704ac1ac36dec59ea8fc05e1a925135c67e Jul 29 20:47:29.997728 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:47:29.997699 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a6ba0b5_5abb_4a5a_b5dd_6e66f25d17d5.slice/crio-5fea9fbbc4e86f25b76041ce6a6bb1c34324a0092ad46ba8ddce7508a4c6f802 WatchSource:0}: Error finding container 5fea9fbbc4e86f25b76041ce6a6bb1c34324a0092ad46ba8ddce7508a4c6f802: Status 404 returned error can't find the container with id 5fea9fbbc4e86f25b76041ce6a6bb1c34324a0092ad46ba8ddce7508a4c6f802 Jul 29 20:47:30.440856 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.440813 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-q7z24\" (UID: \"30987044-0cc1-4e94-91d9-4712292608be\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-q7z24" Jul 29 20:47:30.441345 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.440873 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:47:30.441345 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.440938 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls\") pod \"dns-default-t89sj\" (UID: \"9749b441-c685-4fa6-8044-791cae15b712\") " pod="openshift-dns/dns-default-t89sj" Jul 29 20:47:30.441345 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.441005 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:47:30.441345 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.441044 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-nmw5v\" (UID: \"ffefc971-2b69-4967-bbd3-35da515c68df\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v" Jul 29 20:47:30.441345 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.441069 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-brnnx\" (UID: \"1f181cb2-5f41-425b-97d5-6057df483f86\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" Jul 29 20:47:30.441345 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.441100 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:47:30.442950 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.442908 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-service-ca-bundle\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:47:30.444864 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.444836 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffefc971-2b69-4967-bbd3-35da515c68df-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-nmw5v\" (UID: \"ffefc971-2b69-4967-bbd3-35da515c68df\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v" Jul 29 20:47:30.446718 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.446692 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls\") pod \"image-registry-7f68896cdc-qpf7g\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:47:30.448868 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.448823 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f181cb2-5f41-425b-97d5-6057df483f86-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-brnnx\" (UID: \"1f181cb2-5f41-425b-97d5-6057df483f86\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" Jul 29 20:47:30.450373 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.450350 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9749b441-c685-4fa6-8044-791cae15b712-metrics-tls\") pod \"dns-default-t89sj\" (UID: \"9749b441-c685-4fa6-8044-791cae15b712\") " pod="openshift-dns/dns-default-t89sj" Jul 29 20:47:30.450872 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.450849 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d56bff06-5480-4bcc-bf5b-dbfd2372f9cd-metrics-certs\") pod \"router-default-5f46dc85cf-9562c\" (UID: \"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd\") " pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:47:30.452878 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.452840 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30987044-0cc1-4e94-91d9-4712292608be-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-q7z24\" (UID: \"30987044-0cc1-4e94-91d9-4712292608be\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-q7z24" Jul 29 20:47:30.578104 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.577872 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wf87m\"" Jul 29 20:47:30.587329 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.587139 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:47:30.598831 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.598621 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-p4wrc\"" Jul 29 20:47:30.607818 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.607288 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" Jul 29 20:47:30.609578 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.609556 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cxpqv\"" Jul 29 20:47:30.618390 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.618361 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:47:30.644563 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.644535 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-gncct\"" Jul 29 20:47:30.652315 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.652283 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-79k6t\"" Jul 29 20:47:30.654098 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.653599 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-q7z24" Jul 29 20:47:30.660595 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.660357 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t89sj" Jul 29 20:47:30.681511 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.681484 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-flvjv\"" Jul 29 20:47:30.690307 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.690054 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v" Jul 29 20:47:30.845376 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.844803 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f68896cdc-qpf7g"] Jul 29 20:47:30.876118 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.875679 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx"] Jul 29 20:47:30.888813 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.888315 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7jd4g" event={"ID":"4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5","Type":"ContainerStarted","Data":"ee783c3e6a6ce7039b522a4730a34dc86fc60772167fb08a171ff0e6c4c5b129"} Jul 29 20:47:30.888813 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.888372 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7jd4g" event={"ID":"4a6ba0b5-5abb-4a5a-b5dd-6e66f25d17d5","Type":"ContainerStarted","Data":"5fea9fbbc4e86f25b76041ce6a6bb1c34324a0092ad46ba8ddce7508a4c6f802"} Jul 29 20:47:30.888813 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.888420 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:47:30.890609 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.890540 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" event={"ID":"cc90f71b-6647-4bb2-af21-cfaf42f51393","Type":"ContainerStarted","Data":"2682efc264448889ea663205b3aa0160b2274b993545b33d44c9dab9e0b3fd97"} Jul 29 20:47:30.892170 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.892108 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5sgzv" event={"ID":"347e06fd-9f35-47dd-9442-ddd628e5bd35","Type":"ContainerStarted","Data":"75a9b1aa09b2d682eb5745084cd49704ac1ac36dec59ea8fc05e1a925135c67e"} Jul 29 20:47:30.898178 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.897112 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f46dc85cf-9562c"] Jul 29 20:47:30.911333 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.911276 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7jd4g" podStartSLOduration=67.911252579 podStartE2EDuration="1m7.911252579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:30.909971447 +0000 UTC m=+67.781074914" watchObservedRunningTime="2026-07-29 20:47:30.911252579 +0000 UTC m=+67.782356048" Jul 29 20:47:30.926046 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.924631 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-q7z24"] Jul 29 20:47:30.926046 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.925854 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t89sj"] Jul 29 20:47:30.929500 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:47:30.929470 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9749b441_c685_4fa6_8044_791cae15b712.slice/crio-6130f7ef4f62a19e8bb6e7abf7cbf2bbd13b68fd637c4e20401161d4baf72cae WatchSource:0}: Error finding container 6130f7ef4f62a19e8bb6e7abf7cbf2bbd13b68fd637c4e20401161d4baf72cae: Status 404 returned error can't find the container with id 6130f7ef4f62a19e8bb6e7abf7cbf2bbd13b68fd637c4e20401161d4baf72cae Jul 29 20:47:30.944357 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:47:30.944328 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30987044_0cc1_4e94_91d9_4712292608be.slice/crio-b9c8cf3fab5e98dd49579b29b9743f91dd9c95ba81249cc44c7bf51f9e7daa20 WatchSource:0}: Error finding container b9c8cf3fab5e98dd49579b29b9743f91dd9c95ba81249cc44c7bf51f9e7daa20: Status 404 returned error can't find the container with id b9c8cf3fab5e98dd49579b29b9743f91dd9c95ba81249cc44c7bf51f9e7daa20 Jul 29 20:47:30.958965 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:30.958860 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v"] Jul 29 20:47:31.899950 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:31.899453 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" event={"ID":"cc90f71b-6647-4bb2-af21-cfaf42f51393","Type":"ContainerStarted","Data":"054e03aaf95ceed31e5051e97dd08f3112565da3968f89c0ef5568fb3264c560"} Jul 29 20:47:31.899950 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:31.899692 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:47:31.902028 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:31.901954 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v" event={"ID":"ffefc971-2b69-4967-bbd3-35da515c68df","Type":"ContainerStarted","Data":"95258cd85e6d0d67d7191b86cd1841ba7065ab78837f8c39afdfd079cffff495"} Jul 29 20:47:31.903749 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:31.903677 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t89sj" event={"ID":"9749b441-c685-4fa6-8044-791cae15b712","Type":"ContainerStarted","Data":"6130f7ef4f62a19e8bb6e7abf7cbf2bbd13b68fd637c4e20401161d4baf72cae"} Jul 29 20:47:31.905229 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:31.905184 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-q7z24" event={"ID":"30987044-0cc1-4e94-91d9-4712292608be","Type":"ContainerStarted","Data":"b9c8cf3fab5e98dd49579b29b9743f91dd9c95ba81249cc44c7bf51f9e7daa20"} Jul 29 20:47:31.908963 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:31.908283 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f46dc85cf-9562c" event={"ID":"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd","Type":"ContainerStarted","Data":"d8eb6924222a1fc4e1b658708adb01d9882df647dbebea3e75b6470a9d60847a"} Jul 29 20:47:31.908963 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:31.908419 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f46dc85cf-9562c" event={"ID":"d56bff06-5480-4bcc-bf5b-dbfd2372f9cd","Type":"ContainerStarted","Data":"8938226235e529da97bf55fc0647b9ebf5dd6ad579d1b5883fd74d554f6b8f29"} Jul 29 20:47:31.911205 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:31.911124 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" event={"ID":"1f181cb2-5f41-425b-97d5-6057df483f86","Type":"ContainerStarted","Data":"98de8cfe5f45be05a93ccf7384993129cab951ab03953d4acb3f86580e37e4d1"} Jul 29 20:47:31.919103 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:31.917456 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" podStartSLOduration=67.917437806 podStartE2EDuration="1m7.917437806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:31.916244321 +0000 UTC m=+68.787347789" watchObservedRunningTime="2026-07-29 20:47:31.917437806 +0000 UTC m=+68.788541274" Jul 29 20:47:31.933624 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:31.932904 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5f46dc85cf-9562c" podStartSLOduration=62.932884053 podStartE2EDuration="1m2.932884053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:31.932550935 +0000 UTC m=+68.803654406" watchObservedRunningTime="2026-07-29 20:47:31.932884053 +0000 UTC m=+68.803987521" Jul 29 20:47:32.552117 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:32.552075 2562 scope.go:117] "RemoveContainer" containerID="7664a960dee6ea88886d41ea03b6daa612cae9ce2dc520dfafa93aea302e3351" Jul 29 20:47:32.620267 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:32.618984 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:47:32.622339 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:32.622134 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:47:32.917574 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:32.917508 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:47:32.918456 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:32.918244 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5f46dc85cf-9562c" Jul 29 20:47:38.947038 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:38.946186 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 20:47:38.947038 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:38.946291 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" event={"ID":"bf9e0545-985e-47df-af42-4453c1e24937","Type":"ContainerStarted","Data":"33526e59210dcd4c6040ee917a3f826c9d1e43a641ac8f02ea834bf66c3e0328"} Jul 29 20:47:38.948414 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:38.947622 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:47:38.949667 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:38.949563 2562 patch_prober.go:28] interesting pod/console-operator-584dcb6b54-8v2vf container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Jul 29 20:47:38.949667 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:38.949608 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" podUID="bf9e0545-985e-47df-af42-4453c1e24937" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Jul 29 20:47:38.967711 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:38.966859 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" podStartSLOduration=62.114071672 podStartE2EDuration="1m9.966837729s" totalImagesPullingTime="7.852766057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:38.96503862 +0000 UTC m=+75.836142103" watchObservedRunningTime="2026-07-29 20:47:38.966837729 +0000 UTC m=+75.837941199" Jul 29 20:47:39.955349 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:39.955308 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vgs2s" event={"ID":"053c21f8-14cd-4244-a80d-9713ee979ca3","Type":"ContainerStarted","Data":"53b91e7367e615eb5608e2d88e14ac3a31b6ea6e56a2772d4928b88837fe542c"} Jul 29 20:47:39.957789 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:39.957272 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v" event={"ID":"ffefc971-2b69-4967-bbd3-35da515c68df","Type":"ContainerStarted","Data":"7e8e9ad0abb7e93ea9f4b7d1aad9c96798c7c090425e2d5ff6de0cea0a50a9be"} Jul 29 20:47:39.957789 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:39.957326 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v" event={"ID":"ffefc971-2b69-4967-bbd3-35da515c68df","Type":"ContainerStarted","Data":"0de3730d778d09644998a3200a3a63db69b52e8b801954521657fdce8be29096"} Jul 29 20:47:39.959812 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:39.959731 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t89sj" event={"ID":"9749b441-c685-4fa6-8044-791cae15b712","Type":"ContainerStarted","Data":"2c343d129fde58a926a6d012ee5ed1ac434fbf12173411d758bfbf538e54d483"} Jul 29 20:47:39.959812 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:39.959784 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t89sj" event={"ID":"9749b441-c685-4fa6-8044-791cae15b712","Type":"ContainerStarted","Data":"e6ecb019505bd9a0cb70069018609d2aab5f258414cb796935498dac798a884e"} Jul 29 20:47:39.960026 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:39.959902 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-t89sj" Jul 29 20:47:39.961523 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:39.961497 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-q7z24" event={"ID":"30987044-0cc1-4e94-91d9-4712292608be","Type":"ContainerStarted","Data":"478e464e09b18961ebbd10575c411ed896fc56adc68d0158eb630d7fb5288450"} Jul 29 20:47:39.963418 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:39.963392 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" event={"ID":"1f181cb2-5f41-425b-97d5-6057df483f86","Type":"ContainerStarted","Data":"3f634bd620ef4f6f17a79ab8e35d0f351ed7f34c259fdf21272d81e84dea07c7"} Jul 29 20:47:39.964826 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:39.964800 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fhgt5" event={"ID":"ba4aa585-620a-4c88-8c46-09b343186c70","Type":"ContainerStarted","Data":"4e004afea94277eb77c043deeaa297b82c7b6b8addcc901686157f099b75df42"} Jul 29 20:47:39.966588 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:39.966566 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5sgzv" event={"ID":"347e06fd-9f35-47dd-9442-ddd628e5bd35","Type":"ContainerStarted","Data":"39679095d7cf3a514bbbd0834613c2497b8bbaca0e0b77d77d05d1c35fb948bd"} Jul 29 20:47:39.966675 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:39.966605 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5sgzv" event={"ID":"347e06fd-9f35-47dd-9442-ddd628e5bd35","Type":"ContainerStarted","Data":"f16676a930f423be03087c553ff3fe7355dd5a9ac3c2af1e3bd6f93fdacf270f"} Jul 29 20:47:39.971492 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:39.971473 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-8v2vf" Jul 29 20:47:39.972758 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:39.972712 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-nmw5v" podStartSLOduration=63.181617257 podStartE2EDuration="1m10.972694886s" totalImagesPullingTime="7.791077629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:39.971389079 +0000 UTC m=+76.842492547" watchObservedRunningTime="2026-07-29 20:47:39.972694886 +0000 UTC m=+76.843798351" Jul 29 20:47:39.989085 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:39.989040 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-t89sj" podStartSLOduration=34.115062852 podStartE2EDuration="41.989026148s" totalImagesPullingTime="7.873963296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:39.98752311 +0000 UTC m=+76.858626599" watchObservedRunningTime="2026-07-29 20:47:39.989026148 +0000 UTC m=+76.860129593" Jul 29 20:47:40.003375 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:40.003328 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-q7z24" podStartSLOduration=45.139009785 podStartE2EDuration="53.00331341s" totalImagesPullingTime="7.864303625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:40.001674893 +0000 UTC m=+76.872778361" watchObservedRunningTime="2026-07-29 20:47:40.00331341 +0000 UTC m=+76.874416877" Jul 29 20:47:40.016046 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:40.015994 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fhgt5" podStartSLOduration=67.975420565 podStartE2EDuration="1m17.015978312s" totalImagesPullingTime="9.040557747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:40.01531578 +0000 UTC m=+76.886419249" watchObservedRunningTime="2026-07-29 20:47:40.015978312 +0000 UTC m=+76.887081809" Jul 29 20:47:40.031024 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:40.030978 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5sgzv" podStartSLOduration=71.028950362 podStartE2EDuration="1m17.030958388s" totalImagesPullingTime="6.002008026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:40.029698674 +0000 UTC m=+76.900802141" watchObservedRunningTime="2026-07-29 20:47:40.030958388 +0000 UTC m=+76.902061856" Jul 29 20:47:40.048722 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:40.048658 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-brnnx" podStartSLOduration=63.127145194 podStartE2EDuration="1m11.048637488s" totalImagesPullingTime="7.921492294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:40.048118272 +0000 UTC m=+76.919221739" watchObservedRunningTime="2026-07-29 20:47:40.048637488 +0000 UTC m=+76.919740956" Jul 29 20:47:42.982382 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:42.982353 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vgs2s" event={"ID":"053c21f8-14cd-4244-a80d-9713ee979ca3","Type":"ContainerStarted","Data":"370f6d6e83ff405c79e0f1f491a0306adafc2117bdb85e04157c5210c4d80e74"} Jul 29 20:47:42.999175 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:42.999133 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vgs2s" podStartSLOduration=17.152515129 podStartE2EDuration="34.999118917s" totalImagesPullingTime="17.846603788s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:42.997167889 +0000 UTC m=+79.868271355" watchObservedRunningTime="2026-07-29 20:47:42.999118917 +0000 UTC m=+79.870222383" Jul 29 20:47:49.721354 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.721237 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57f5bd7f55-2s8kk"] Jul 29 20:47:49.724123 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.724100 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:49.726080 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.726054 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 29 20:47:49.726197 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.726054 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 29 20:47:49.726452 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.726429 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7g22t\"" Jul 29 20:47:49.726552 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.726471 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 29 20:47:49.726552 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.726477 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 29 20:47:49.726671 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.726621 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 29 20:47:49.726843 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.726828 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 29 20:47:49.726971 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.726956 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 29 20:47:49.734596 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.734571 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57f5bd7f55-2s8kk"] Jul 29 20:47:49.797561 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.797525 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7119b239-5157-44f0-aecc-1fef1d56a20b-console-oauth-config\") pod \"console-57f5bd7f55-2s8kk\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:49.797561 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.797562 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7119b239-5157-44f0-aecc-1fef1d56a20b-console-config\") pod \"console-57f5bd7f55-2s8kk\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:49.797780 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.797582 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7119b239-5157-44f0-aecc-1fef1d56a20b-oauth-serving-cert\") pod \"console-57f5bd7f55-2s8kk\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:49.797780 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.797672 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7119b239-5157-44f0-aecc-1fef1d56a20b-service-ca\") pod \"console-57f5bd7f55-2s8kk\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:49.797780 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.797704 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7119b239-5157-44f0-aecc-1fef1d56a20b-console-serving-cert\") pod \"console-57f5bd7f55-2s8kk\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:49.797780 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.797724 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4px9h\" (UniqueName: \"kubernetes.io/projected/7119b239-5157-44f0-aecc-1fef1d56a20b-kube-api-access-4px9h\") pod \"console-57f5bd7f55-2s8kk\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:49.898097 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.898059 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7119b239-5157-44f0-aecc-1fef1d56a20b-console-oauth-config\") pod \"console-57f5bd7f55-2s8kk\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:49.898097 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.898098 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7119b239-5157-44f0-aecc-1fef1d56a20b-console-config\") pod \"console-57f5bd7f55-2s8kk\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:49.898316 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.898116 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7119b239-5157-44f0-aecc-1fef1d56a20b-oauth-serving-cert\") pod \"console-57f5bd7f55-2s8kk\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:49.898316 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.898175 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7119b239-5157-44f0-aecc-1fef1d56a20b-service-ca\") pod \"console-57f5bd7f55-2s8kk\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:49.898316 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.898201 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7119b239-5157-44f0-aecc-1fef1d56a20b-console-serving-cert\") pod \"console-57f5bd7f55-2s8kk\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:49.898316 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.898224 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4px9h\" (UniqueName: \"kubernetes.io/projected/7119b239-5157-44f0-aecc-1fef1d56a20b-kube-api-access-4px9h\") pod \"console-57f5bd7f55-2s8kk\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:49.898832 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.898810 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7119b239-5157-44f0-aecc-1fef1d56a20b-console-config\") pod \"console-57f5bd7f55-2s8kk\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:49.898956 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.898835 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7119b239-5157-44f0-aecc-1fef1d56a20b-oauth-serving-cert\") pod \"console-57f5bd7f55-2s8kk\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:49.898956 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.898887 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7119b239-5157-44f0-aecc-1fef1d56a20b-service-ca\") pod \"console-57f5bd7f55-2s8kk\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:49.900557 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.900532 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7119b239-5157-44f0-aecc-1fef1d56a20b-console-serving-cert\") pod \"console-57f5bd7f55-2s8kk\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:49.900557 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.900540 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7119b239-5157-44f0-aecc-1fef1d56a20b-console-oauth-config\") pod \"console-57f5bd7f55-2s8kk\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:49.905668 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.905647 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4px9h\" (UniqueName: \"kubernetes.io/projected/7119b239-5157-44f0-aecc-1fef1d56a20b-kube-api-access-4px9h\") pod \"console-57f5bd7f55-2s8kk\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:49.978104 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:49.978025 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t89sj" Jul 29 20:47:50.033872 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.033837 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:47:50.159800 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.159766 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57f5bd7f55-2s8kk"] Jul 29 20:47:50.162604 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:47:50.162577 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7119b239_5157_44f0_aecc_1fef1d56a20b.slice/crio-a024804ba4ef4eaa07b8470d683ac7e33510894875e1af791d20bf1bc8f1546c WatchSource:0}: Error finding container a024804ba4ef4eaa07b8470d683ac7e33510894875e1af791d20bf1bc8f1546c: Status 404 returned error can't find the container with id a024804ba4ef4eaa07b8470d683ac7e33510894875e1af791d20bf1bc8f1546c Jul 29 20:47:50.591732 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.591693 2562 patch_prober.go:28] interesting pod/image-registry-7f68896cdc-qpf7g container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 20:47:50.591912 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.591748 2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" podUID="cc90f71b-6647-4bb2-af21-cfaf42f51393" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 20:47:50.846547 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.846459 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7l48v"] Jul 29 20:47:50.849427 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.849397 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg"] Jul 29 20:47:50.849882 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.849858 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:50.852095 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.852057 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 20:47:50.852206 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.852164 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:50.852644 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.852621 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 20:47:50.853619 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.853026 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 20:47:50.853619 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.853058 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 20:47:50.853619 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.853234 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xmffz\"" Jul 29 20:47:50.854078 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.853979 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 29 20:47:50.854577 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.854352 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 29 20:47:50.854577 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.854360 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 29 20:47:50.854830 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.854810 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rs4f4\"" Jul 29 20:47:50.861913 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.861891 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg"] Jul 29 20:47:50.905891 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.905854 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2ffa2c61-2436-4157-8edc-a5f360c28d8b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-rqpwg\" (UID: \"2ffa2c61-2436-4157-8edc-a5f360c28d8b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:50.906096 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.905900 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26cbdcfd-b96e-405f-912c-db5b721a7f91-sys\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:50.906096 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.905962 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/26cbdcfd-b96e-405f-912c-db5b721a7f91-node-exporter-accelerators-collector-config\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:50.906096 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.905989 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26cbdcfd-b96e-405f-912c-db5b721a7f91-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:50.906096 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.906014 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/26cbdcfd-b96e-405f-912c-db5b721a7f91-node-exporter-textfile\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:50.906096 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.906043 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/26cbdcfd-b96e-405f-912c-db5b721a7f91-node-exporter-wtmp\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:50.906096 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.906064 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/26cbdcfd-b96e-405f-912c-db5b721a7f91-node-exporter-tls\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:50.906096 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.906088 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26cbdcfd-b96e-405f-912c-db5b721a7f91-metrics-client-ca\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:50.906452 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.906111 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rrwg\" (UniqueName: \"kubernetes.io/projected/2ffa2c61-2436-4157-8edc-a5f360c28d8b-kube-api-access-8rrwg\") pod \"kube-state-metrics-56d9f944b4-rqpwg\" (UID: \"2ffa2c61-2436-4157-8edc-a5f360c28d8b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:50.906452 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.906136 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2ffa2c61-2436-4157-8edc-a5f360c28d8b-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-rqpwg\" (UID: \"2ffa2c61-2436-4157-8edc-a5f360c28d8b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:50.906452 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.906169 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ffa2c61-2436-4157-8edc-a5f360c28d8b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-rqpwg\" (UID: \"2ffa2c61-2436-4157-8edc-a5f360c28d8b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:50.906452 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.906198 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlhrb\" (UniqueName: \"kubernetes.io/projected/26cbdcfd-b96e-405f-912c-db5b721a7f91-kube-api-access-zlhrb\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:50.906452 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.906222 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ffa2c61-2436-4157-8edc-a5f360c28d8b-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-rqpwg\" (UID: \"2ffa2c61-2436-4157-8edc-a5f360c28d8b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:50.906452 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.906247 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ffa2c61-2436-4157-8edc-a5f360c28d8b-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-rqpwg\" (UID: \"2ffa2c61-2436-4157-8edc-a5f360c28d8b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:50.906452 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:50.906323 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/26cbdcfd-b96e-405f-912c-db5b721a7f91-root\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.007463 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.007420 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/26cbdcfd-b96e-405f-912c-db5b721a7f91-node-exporter-accelerators-collector-config\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.007642 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.007474 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26cbdcfd-b96e-405f-912c-db5b721a7f91-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.007642 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.007505 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/26cbdcfd-b96e-405f-912c-db5b721a7f91-node-exporter-textfile\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.007642 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.007533 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/26cbdcfd-b96e-405f-912c-db5b721a7f91-node-exporter-wtmp\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.007642 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.007557 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/26cbdcfd-b96e-405f-912c-db5b721a7f91-node-exporter-tls\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.007642 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.007579 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26cbdcfd-b96e-405f-912c-db5b721a7f91-metrics-client-ca\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.007642 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.007601 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rrwg\" (UniqueName: \"kubernetes.io/projected/2ffa2c61-2436-4157-8edc-a5f360c28d8b-kube-api-access-8rrwg\") pod \"kube-state-metrics-56d9f944b4-rqpwg\" (UID: \"2ffa2c61-2436-4157-8edc-a5f360c28d8b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:51.007642 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.007627 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2ffa2c61-2436-4157-8edc-a5f360c28d8b-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-rqpwg\" (UID: \"2ffa2c61-2436-4157-8edc-a5f360c28d8b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:51.008020 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.007653 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ffa2c61-2436-4157-8edc-a5f360c28d8b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-rqpwg\" (UID: \"2ffa2c61-2436-4157-8edc-a5f360c28d8b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:51.008020 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.007678 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlhrb\" (UniqueName: \"kubernetes.io/projected/26cbdcfd-b96e-405f-912c-db5b721a7f91-kube-api-access-zlhrb\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.008020 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.007700 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ffa2c61-2436-4157-8edc-a5f360c28d8b-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-rqpwg\" (UID: \"2ffa2c61-2436-4157-8edc-a5f360c28d8b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:51.008020 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.007724 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ffa2c61-2436-4157-8edc-a5f360c28d8b-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-rqpwg\" (UID: \"2ffa2c61-2436-4157-8edc-a5f360c28d8b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:51.008020 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.007753 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/26cbdcfd-b96e-405f-912c-db5b721a7f91-node-exporter-wtmp\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.008020 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.007756 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/26cbdcfd-b96e-405f-912c-db5b721a7f91-root\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.008020 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.007791 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/26cbdcfd-b96e-405f-912c-db5b721a7f91-root\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.008020 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.007867 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2ffa2c61-2436-4157-8edc-a5f360c28d8b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-rqpwg\" (UID: \"2ffa2c61-2436-4157-8edc-a5f360c28d8b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:51.008020 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.007897 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26cbdcfd-b96e-405f-912c-db5b721a7f91-sys\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.008020 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.008019 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26cbdcfd-b96e-405f-912c-db5b721a7f91-sys\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.008558 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:51.008108 2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 29 20:47:51.008558 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.008121 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/26cbdcfd-b96e-405f-912c-db5b721a7f91-node-exporter-accelerators-collector-config\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.008558 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:51.008170 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cbdcfd-b96e-405f-912c-db5b721a7f91-node-exporter-tls podName:26cbdcfd-b96e-405f-912c-db5b721a7f91 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:51.50814763 +0000 UTC m=+88.379251089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/26cbdcfd-b96e-405f-912c-db5b721a7f91-node-exporter-tls") pod "node-exporter-7l48v" (UID: "26cbdcfd-b96e-405f-912c-db5b721a7f91") : secret "node-exporter-tls" not found Jul 29 20:47:51.008687 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.008634 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26cbdcfd-b96e-405f-912c-db5b721a7f91-metrics-client-ca\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.008763 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.008734 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/26cbdcfd-b96e-405f-912c-db5b721a7f91-node-exporter-textfile\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.009164 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.008907 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ffa2c61-2436-4157-8edc-a5f360c28d8b-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-rqpwg\" (UID: \"2ffa2c61-2436-4157-8edc-a5f360c28d8b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:51.009164 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:51.009056 2562 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 29 20:47:51.009164 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:47:51.009107 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ffa2c61-2436-4157-8edc-a5f360c28d8b-kube-state-metrics-tls podName:2ffa2c61-2436-4157-8edc-a5f360c28d8b nodeName:}" failed. No retries permitted until 2026-07-29 20:47:51.509089856 +0000 UTC m=+88.380193313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/2ffa2c61-2436-4157-8edc-a5f360c28d8b-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-rqpwg" (UID: "2ffa2c61-2436-4157-8edc-a5f360c28d8b") : secret "kube-state-metrics-tls" not found Jul 29 20:47:51.009437 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.009195 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2ffa2c61-2436-4157-8edc-a5f360c28d8b-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-rqpwg\" (UID: \"2ffa2c61-2436-4157-8edc-a5f360c28d8b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:51.009437 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.009420 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2ffa2c61-2436-4157-8edc-a5f360c28d8b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-rqpwg\" (UID: \"2ffa2c61-2436-4157-8edc-a5f360c28d8b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:51.010947 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.010857 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26cbdcfd-b96e-405f-912c-db5b721a7f91-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.011974 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.011930 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ffa2c61-2436-4157-8edc-a5f360c28d8b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-rqpwg\" (UID: \"2ffa2c61-2436-4157-8edc-a5f360c28d8b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:51.013637 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.013557 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f5bd7f55-2s8kk" event={"ID":"7119b239-5157-44f0-aecc-1fef1d56a20b","Type":"ContainerStarted","Data":"a024804ba4ef4eaa07b8470d683ac7e33510894875e1af791d20bf1bc8f1546c"} Jul 29 20:47:51.016068 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.016026 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlhrb\" (UniqueName: \"kubernetes.io/projected/26cbdcfd-b96e-405f-912c-db5b721a7f91-kube-api-access-zlhrb\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.016862 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.016837 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rrwg\" (UniqueName: \"kubernetes.io/projected/2ffa2c61-2436-4157-8edc-a5f360c28d8b-kube-api-access-8rrwg\") pod \"kube-state-metrics-56d9f944b4-rqpwg\" (UID: \"2ffa2c61-2436-4157-8edc-a5f360c28d8b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:51.513907 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.513869 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/26cbdcfd-b96e-405f-912c-db5b721a7f91-node-exporter-tls\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.514123 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.513927 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ffa2c61-2436-4157-8edc-a5f360c28d8b-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-rqpwg\" (UID: \"2ffa2c61-2436-4157-8edc-a5f360c28d8b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:51.516821 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.516795 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/26cbdcfd-b96e-405f-912c-db5b721a7f91-node-exporter-tls\") pod \"node-exporter-7l48v\" (UID: \"26cbdcfd-b96e-405f-912c-db5b721a7f91\") " pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.516957 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.516866 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ffa2c61-2436-4157-8edc-a5f360c28d8b-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-rqpwg\" (UID: \"2ffa2c61-2436-4157-8edc-a5f360c28d8b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:51.766002 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.765895 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7l48v" Jul 29 20:47:51.774149 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.774063 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" Jul 29 20:47:51.899465 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.899420 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 20:47:51.903036 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.903010 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:51.905097 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.904934 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 29 20:47:51.905097 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.904934 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 29 20:47:51.905641 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.905341 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 29 20:47:51.905641 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.905352 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 29 20:47:51.905641 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.905426 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 29 20:47:51.905641 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.905357 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 29 20:47:51.905870 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.905661 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 29 20:47:51.905870 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.905710 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 29 20:47:51.905870 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.905833 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wzfqg\"" Jul 29 20:47:51.906050 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.905910 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 29 20:47:51.916533 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.916444 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 20:47:51.933877 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:51.933847 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg"] Jul 29 20:47:51.936452 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:47:51.936422 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffa2c61_2436_4157_8edc_a5f360c28d8b.slice/crio-8e20736e3a7912ee445329dea6f18fabda6c0fb1911a1452041576535787182b WatchSource:0}: Error finding container 8e20736e3a7912ee445329dea6f18fabda6c0fb1911a1452041576535787182b: Status 404 returned error can't find the container with id 8e20736e3a7912ee445329dea6f18fabda6c0fb1911a1452041576535787182b Jul 29 20:47:52.018615 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.018525 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k9lj\" (UniqueName: \"kubernetes.io/projected/01ec765f-ea51-44ac-b38c-5eb565d63f6f-kube-api-access-2k9lj\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.018615 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.018594 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.018824 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.018629 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01ec765f-ea51-44ac-b38c-5eb565d63f6f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.018824 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.018668 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/01ec765f-ea51-44ac-b38c-5eb565d63f6f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.018824 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.018710 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ec765f-ea51-44ac-b38c-5eb565d63f6f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.018824 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.018759 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.018824 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.018789 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-web-config\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.019087 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.018825 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.019087 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.018861 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01ec765f-ea51-44ac-b38c-5eb565d63f6f-config-out\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.019087 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.018893 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-config-volume\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.019087 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.018968 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.019087 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.018992 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01ec765f-ea51-44ac-b38c-5eb565d63f6f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.019087 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.019040 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.026157 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.026106 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" event={"ID":"2ffa2c61-2436-4157-8edc-a5f360c28d8b","Type":"ContainerStarted","Data":"8e20736e3a7912ee445329dea6f18fabda6c0fb1911a1452041576535787182b"} Jul 29 20:47:52.027503 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.027465 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7l48v" event={"ID":"26cbdcfd-b96e-405f-912c-db5b721a7f91","Type":"ContainerStarted","Data":"e08c02dcc9145ba6c0c9fde940e6489b3a938db8eaf931599b1097c9cb997258"} Jul 29 20:47:52.120634 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.120594 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.120788 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.120651 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k9lj\" (UniqueName: \"kubernetes.io/projected/01ec765f-ea51-44ac-b38c-5eb565d63f6f-kube-api-access-2k9lj\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.120788 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.120698 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.120788 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.120726 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01ec765f-ea51-44ac-b38c-5eb565d63f6f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.120788 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.120751 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/01ec765f-ea51-44ac-b38c-5eb565d63f6f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.120788 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.120776 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ec765f-ea51-44ac-b38c-5eb565d63f6f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.121072 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.120814 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.121072 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.120840 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-web-config\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.121072 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.120865 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.121072 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.120894 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01ec765f-ea51-44ac-b38c-5eb565d63f6f-config-out\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.121072 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.120944 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-config-volume\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.121072 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.120996 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.121072 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.121019 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01ec765f-ea51-44ac-b38c-5eb565d63f6f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.121462 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.121436 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/01ec765f-ea51-44ac-b38c-5eb565d63f6f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.122238 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.122208 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01ec765f-ea51-44ac-b38c-5eb565d63f6f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.123686 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.123330 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ec765f-ea51-44ac-b38c-5eb565d63f6f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.124363 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.124168 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-web-config\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.126203 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.124592 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.126683 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.126639 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.126803 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.126776 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01ec765f-ea51-44ac-b38c-5eb565d63f6f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.128218 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.127143 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.128218 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.127493 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.128218 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.128163 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01ec765f-ea51-44ac-b38c-5eb565d63f6f-config-out\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.128575 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.128535 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.128768 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.128740 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-config-volume\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.131163 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.131122 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k9lj\" (UniqueName: \"kubernetes.io/projected/01ec765f-ea51-44ac-b38c-5eb565d63f6f-kube-api-access-2k9lj\") pod \"alertmanager-main-0\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:52.210084 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.210045 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f68896cdc-qpf7g"] Jul 29 20:47:52.216262 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.216192 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:47:52.217883 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:52.217649 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:47:53.596497 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:53.596389 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 20:47:53.599328 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:47:53.599287 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ec765f_ea51_44ac_b38c_5eb565d63f6f.slice/crio-143548c4be64977b6d791599d2cb5727648428ce915293995b8f4fab3c9229c2 WatchSource:0}: Error finding container 143548c4be64977b6d791599d2cb5727648428ce915293995b8f4fab3c9229c2: Status 404 returned error can't find the container with id 143548c4be64977b6d791599d2cb5727648428ce915293995b8f4fab3c9229c2 Jul 29 20:47:54.038114 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:54.038078 2562 generic.go:356] "Generic (PLEG): container finished" podID="26cbdcfd-b96e-405f-912c-db5b721a7f91" containerID="d6ad25b2e42151a09292bb6bfc78698adf8e96a6623f637911713d7a2c74baa5" exitCode=0 Jul 29 20:47:54.038284 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:54.038161 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7l48v" event={"ID":"26cbdcfd-b96e-405f-912c-db5b721a7f91","Type":"ContainerDied","Data":"d6ad25b2e42151a09292bb6bfc78698adf8e96a6623f637911713d7a2c74baa5"} Jul 29 20:47:54.039523 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:54.039493 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01ec765f-ea51-44ac-b38c-5eb565d63f6f","Type":"ContainerStarted","Data":"143548c4be64977b6d791599d2cb5727648428ce915293995b8f4fab3c9229c2"} Jul 29 20:47:54.041258 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:54.041225 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f5bd7f55-2s8kk" event={"ID":"7119b239-5157-44f0-aecc-1fef1d56a20b","Type":"ContainerStarted","Data":"35929220d890ccae5f1b84f33b3c1b65a3b84cffed9cc8753f144efc8c6b40f9"} Jul 29 20:47:54.070581 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:54.070425 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57f5bd7f55-2s8kk" podStartSLOduration=1.796241602 podStartE2EDuration="5.070406495s" totalImagesPullingTime="3.274164893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:54.070377159 +0000 UTC m=+90.941480629" watchObservedRunningTime="2026-07-29 20:47:54.070406495 +0000 UTC m=+90.941509964" Jul 29 20:47:55.048613 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.048577 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7l48v" event={"ID":"26cbdcfd-b96e-405f-912c-db5b721a7f91","Type":"ContainerStarted","Data":"368be655ef40de1d993be2e0eabbeefd84a504057b4eb5da8e878d84491c8702"} Jul 29 20:47:55.049049 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.048637 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7l48v" event={"ID":"26cbdcfd-b96e-405f-912c-db5b721a7f91","Type":"ContainerStarted","Data":"9b448149c4aa8a16059838a8a657236fc211f0cec604be1e4e9bf20a17d41e77"} Jul 29 20:47:55.050286 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.050251 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01ec765f-ea51-44ac-b38c-5eb565d63f6f","Type":"ContainerStarted","Data":"b5cd54c95996e2f0b1290fe446ddaf0349ca22dece2cc86dd1aaf966ebd5c29b"} Jul 29 20:47:55.053065 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.052939 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" event={"ID":"2ffa2c61-2436-4157-8edc-a5f360c28d8b","Type":"ContainerStarted","Data":"e76e9fa83714cccb4ddbfe0297b788dfbab9247862c940d2e62677f4c0bbe9a3"} Jul 29 20:47:55.053065 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.052994 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" event={"ID":"2ffa2c61-2436-4157-8edc-a5f360c28d8b","Type":"ContainerStarted","Data":"5cd54f4e3959c57ab096709cdc59a46dbf80eaebbd4c2cd020de7ed304005066"} Jul 29 20:47:55.053065 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.053013 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" event={"ID":"2ffa2c61-2436-4157-8edc-a5f360c28d8b","Type":"ContainerStarted","Data":"cc086531147809be6372c4bda3e4ce8575922350dbab151aa5facd201607ee35"} Jul 29 20:47:55.066615 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.066572 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7l48v" podStartSLOduration=3.325667853 podStartE2EDuration="5.066557464s" totalImagesPullingTime="1.654680338s" totalInitContainerRuntime="86.209273ms" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:55.065049985 +0000 UTC m=+91.936153453" watchObservedRunningTime="2026-07-29 20:47:55.066557464 +0000 UTC m=+91.937660931" Jul 29 20:47:55.084203 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.084159 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rqpwg" podStartSLOduration=2.765486538 podStartE2EDuration="5.084145382s" totalImagesPullingTime="2.318658844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:55.082584272 +0000 UTC m=+91.953687740" watchObservedRunningTime="2026-07-29 20:47:55.084145382 +0000 UTC m=+91.955248850" Jul 29 20:47:55.134478 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.134442 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-577c88c6f7-qb7xj"] Jul 29 20:47:55.136735 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.136714 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.139127 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.139109 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 29 20:47:55.139544 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.139509 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 29 20:47:55.139654 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.139579 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-86ka09pfimvtr\"" Jul 29 20:47:55.139802 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.139789 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 29 20:47:55.140309 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.140292 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 29 20:47:55.140353 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.140313 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rfrl5\"" Jul 29 20:47:55.149500 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.149475 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-577c88c6f7-qb7xj"] Jul 29 20:47:55.248741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.248641 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/14d57622-e026-4e5e-8608-16450670ad62-secret-metrics-server-client-certs\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.248741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.248684 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/14d57622-e026-4e5e-8608-16450670ad62-client-ca-bundle\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.248741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.248723 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/14d57622-e026-4e5e-8608-16450670ad62-secret-metrics-server-tls\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.249052 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.248759 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlng9\" (UniqueName: \"kubernetes.io/projected/14d57622-e026-4e5e-8608-16450670ad62-kube-api-access-qlng9\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.249052 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.248811 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d57622-e026-4e5e-8608-16450670ad62-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.249052 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.248876 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/14d57622-e026-4e5e-8608-16450670ad62-metrics-server-audit-profiles\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.249052 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.248911 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/14d57622-e026-4e5e-8608-16450670ad62-audit-log\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.349706 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.349655 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/14d57622-e026-4e5e-8608-16450670ad62-secret-metrics-server-client-certs\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.349706 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.349703 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/14d57622-e026-4e5e-8608-16450670ad62-client-ca-bundle\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.349988 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.349743 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/14d57622-e026-4e5e-8608-16450670ad62-secret-metrics-server-tls\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.349988 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.349767 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlng9\" (UniqueName: \"kubernetes.io/projected/14d57622-e026-4e5e-8608-16450670ad62-kube-api-access-qlng9\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.349988 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.349802 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d57622-e026-4e5e-8608-16450670ad62-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.349988 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.349870 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/14d57622-e026-4e5e-8608-16450670ad62-metrics-server-audit-profiles\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.349988 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.349910 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/14d57622-e026-4e5e-8608-16450670ad62-audit-log\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.350779 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.350329 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/14d57622-e026-4e5e-8608-16450670ad62-audit-log\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.350779 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.350712 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d57622-e026-4e5e-8608-16450670ad62-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.351113 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.351088 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/14d57622-e026-4e5e-8608-16450670ad62-metrics-server-audit-profiles\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.352249 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.352222 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/14d57622-e026-4e5e-8608-16450670ad62-client-ca-bundle\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.352316 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.352303 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/14d57622-e026-4e5e-8608-16450670ad62-secret-metrics-server-tls\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.352362 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.352348 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/14d57622-e026-4e5e-8608-16450670ad62-secret-metrics-server-client-certs\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.356730 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.356714 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlng9\" (UniqueName: \"kubernetes.io/projected/14d57622-e026-4e5e-8608-16450670ad62-kube-api-access-qlng9\") pod \"metrics-server-577c88c6f7-qb7xj\" (UID: \"14d57622-e026-4e5e-8608-16450670ad62\") " pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.446110 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.446073 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:47:55.572643 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:55.572608 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-577c88c6f7-qb7xj"] Jul 29 20:47:55.576292 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:47:55.576258 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d57622_e026_4e5e_8608_16450670ad62.slice/crio-58d88bbfda81def75eeb657b0a5fa53c9aefef31d8e7e593743c336a9d4a7140 WatchSource:0}: Error finding container 58d88bbfda81def75eeb657b0a5fa53c9aefef31d8e7e593743c336a9d4a7140: Status 404 returned error can't find the container with id 58d88bbfda81def75eeb657b0a5fa53c9aefef31d8e7e593743c336a9d4a7140 Jul 29 20:47:56.059508 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:56.059473 2562 generic.go:356] "Generic (PLEG): container finished" podID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerID="b5cd54c95996e2f0b1290fe446ddaf0349ca22dece2cc86dd1aaf966ebd5c29b" exitCode=0 Jul 29 20:47:56.059973 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:56.059565 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01ec765f-ea51-44ac-b38c-5eb565d63f6f","Type":"ContainerDied","Data":"b5cd54c95996e2f0b1290fe446ddaf0349ca22dece2cc86dd1aaf966ebd5c29b"} Jul 29 20:47:56.060979 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:56.060952 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" event={"ID":"14d57622-e026-4e5e-8608-16450670ad62","Type":"ContainerStarted","Data":"58d88bbfda81def75eeb657b0a5fa53c9aefef31d8e7e593743c336a9d4a7140"} Jul 29 20:47:58.073301 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.073259 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01ec765f-ea51-44ac-b38c-5eb565d63f6f","Type":"ContainerStarted","Data":"0b737f2f659b6aa832474270e109c6dcf82980e5d43cd9c0f5dfab63e5d3db6b"} Jul 29 20:47:58.073730 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.073356 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01ec765f-ea51-44ac-b38c-5eb565d63f6f","Type":"ContainerStarted","Data":"a00cb10010c66e4afc640e30b06abff6d2c8986c049fdda13fcacf6f48540480"} Jul 29 20:47:58.073730 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.073403 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01ec765f-ea51-44ac-b38c-5eb565d63f6f","Type":"ContainerStarted","Data":"84e7f355a92d174b8b898653580e9551bc8f788e52f6e70c234f70d814fd374d"} Jul 29 20:47:58.073730 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.073442 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01ec765f-ea51-44ac-b38c-5eb565d63f6f","Type":"ContainerStarted","Data":"8327c154ba2f4110332b0a54274de9aa3ea4c64fbc11bd07b00a4ebed5b58941"} Jul 29 20:47:58.073730 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.073479 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01ec765f-ea51-44ac-b38c-5eb565d63f6f","Type":"ContainerStarted","Data":"cc79d8eb057249931dc8707c73aa3faad29d2d14b1e46301ef658c09bef1072a"} Jul 29 20:47:58.074737 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.074704 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" event={"ID":"14d57622-e026-4e5e-8608-16450670ad62","Type":"ContainerStarted","Data":"188c88e781f3f8bef97c2b029dff0f0400afa0e8484dbce1248bfdee79320fe9"} Jul 29 20:47:58.093289 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.093240 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" podStartSLOduration=1.083562633 podStartE2EDuration="3.093224613s" totalImagesPullingTime="2.00966198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:58.091333722 +0000 UTC m=+94.962437189" watchObservedRunningTime="2026-07-29 20:47:58.093224613 +0000 UTC m=+94.964328080" Jul 29 20:47:58.319116 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.319082 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bb6b5b74f-wrrwg"] Jul 29 20:47:58.321422 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.321404 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.330582 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.329057 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 29 20:47:58.330582 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.330363 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bb6b5b74f-wrrwg"] Jul 29 20:47:58.375495 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.375457 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-trusted-ca-bundle\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.375693 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.375501 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fefe1fd2-a131-4c08-bbc6-2d634e86467c-console-oauth-config\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.375693 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.375603 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-console-config\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.375693 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.375643 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58sv8\" (UniqueName: \"kubernetes.io/projected/fefe1fd2-a131-4c08-bbc6-2d634e86467c-kube-api-access-58sv8\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.375833 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.375697 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-oauth-serving-cert\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.375833 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.375716 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-service-ca\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.375833 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.375783 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fefe1fd2-a131-4c08-bbc6-2d634e86467c-console-serving-cert\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.476642 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.476606 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-trusted-ca-bundle\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.476817 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.476778 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fefe1fd2-a131-4c08-bbc6-2d634e86467c-console-oauth-config\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.476881 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.476838 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-console-config\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.476881 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.476864 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58sv8\" (UniqueName: \"kubernetes.io/projected/fefe1fd2-a131-4c08-bbc6-2d634e86467c-kube-api-access-58sv8\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.477008 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.476911 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-oauth-serving-cert\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.477008 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.476963 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-service-ca\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.477110 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.477017 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fefe1fd2-a131-4c08-bbc6-2d634e86467c-console-serving-cert\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.477569 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.477541 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-trusted-ca-bundle\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.477791 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.477739 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-oauth-serving-cert\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.477791 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.477778 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-console-config\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.478002 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.477799 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-service-ca\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.479670 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.479646 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fefe1fd2-a131-4c08-bbc6-2d634e86467c-console-oauth-config\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.479796 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.479773 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fefe1fd2-a131-4c08-bbc6-2d634e86467c-console-serving-cert\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.485103 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.485081 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58sv8\" (UniqueName: \"kubernetes.io/projected/fefe1fd2-a131-4c08-bbc6-2d634e86467c-kube-api-access-58sv8\") pod \"console-5bb6b5b74f-wrrwg\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.632597 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.632505 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:47:58.842884 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:58.842857 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bb6b5b74f-wrrwg"] Jul 29 20:47:58.845176 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:47:58.845152 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefe1fd2_a131_4c08_bbc6_2d634e86467c.slice/crio-3a419e437e01a78b49a8edaaa8df6856a877d5fcb8de886999c7f7afcb5a1c4c WatchSource:0}: Error finding container 3a419e437e01a78b49a8edaaa8df6856a877d5fcb8de886999c7f7afcb5a1c4c: Status 404 returned error can't find the container with id 3a419e437e01a78b49a8edaaa8df6856a877d5fcb8de886999c7f7afcb5a1c4c Jul 29 20:47:59.082410 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:59.082376 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01ec765f-ea51-44ac-b38c-5eb565d63f6f","Type":"ContainerStarted","Data":"3b40e92336cf7b8b5104dec082302f6b08466ead39b6b9998e9249dbe0312ff9"} Jul 29 20:47:59.084048 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:59.084020 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bb6b5b74f-wrrwg" event={"ID":"fefe1fd2-a131-4c08-bbc6-2d634e86467c","Type":"ContainerStarted","Data":"e89bcadd1b0b045cb0a66617c85bb36dedd500a0ca8cab8eb6b6d869799bc81e"} Jul 29 20:47:59.084189 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:59.084064 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bb6b5b74f-wrrwg" event={"ID":"fefe1fd2-a131-4c08-bbc6-2d634e86467c","Type":"ContainerStarted","Data":"3a419e437e01a78b49a8edaaa8df6856a877d5fcb8de886999c7f7afcb5a1c4c"} Jul 29 20:47:59.107742 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:59.107687 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.370814107 podStartE2EDuration="8.107670215s" totalImagesPullingTime="3.719947397s" totalInitContainerRuntime="16.908711ms" isStatefulPod=false podCreationTimestamp="2026-07-29 20:47:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:59.106231417 +0000 UTC m=+95.977334903" watchObservedRunningTime="2026-07-29 20:47:59.107670215 +0000 UTC m=+95.978773681" Jul 29 20:47:59.123320 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:47:59.123271 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bb6b5b74f-wrrwg" podStartSLOduration=1.123255439 podStartE2EDuration="1.123255439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:59.121579556 +0000 UTC m=+95.992683025" watchObservedRunningTime="2026-07-29 20:47:59.123255439 +0000 UTC m=+95.994358906" Jul 29 20:48:00.034205 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:00.034161 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:48:00.034394 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:00.034231 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:48:00.038908 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:00.038877 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:48:00.092155 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:00.092124 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:48:01.824193 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:01.824150 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bb6b5b74f-wrrwg"] Jul 29 20:48:01.851969 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:01.851934 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66784977cd-dw2zf"] Jul 29 20:48:01.901776 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:01.901736 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66784977cd-dw2zf"] Jul 29 20:48:01.901956 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:01.901840 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:01.913275 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:01.913248 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7jd4g" Jul 29 20:48:02.010124 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.010079 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-service-ca\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.010263 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.010128 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l9bp\" (UniqueName: \"kubernetes.io/projected/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-kube-api-access-2l9bp\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.010263 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.010246 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-trusted-ca-bundle\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.010326 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.010279 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-console-serving-cert\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.010370 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.010346 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-oauth-serving-cert\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.010424 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.010382 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-console-oauth-config\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.010424 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.010419 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-console-config\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.111262 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.111173 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-service-ca\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.111262 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.111212 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l9bp\" (UniqueName: \"kubernetes.io/projected/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-kube-api-access-2l9bp\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.111262 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.111240 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-trusted-ca-bundle\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.111262 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.111261 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-console-serving-cert\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.111592 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.111315 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-oauth-serving-cert\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.111592 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.111361 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-console-oauth-config\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.111592 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.111399 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-console-config\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.112108 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.112082 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-service-ca\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.112199 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.112154 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-console-config\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.112233 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.112199 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-oauth-serving-cert\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.112264 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.112241 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-trusted-ca-bundle\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.113865 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.113846 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-console-serving-cert\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.113990 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.113970 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-console-oauth-config\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.118154 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.118135 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l9bp\" (UniqueName: \"kubernetes.io/projected/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-kube-api-access-2l9bp\") pod \"console-66784977cd-dw2zf\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.211020 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.210983 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:02.401549 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:02.401459 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66784977cd-dw2zf"] Jul 29 20:48:02.405600 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:48:02.405564 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f20f2ec_d69d_47cd_8948_0e78ba4c0995.slice/crio-677e4827d4ea856d15e4e4839f4c4f557d6eeb39c11023d668881b9ecfbff48f WatchSource:0}: Error finding container 677e4827d4ea856d15e4e4839f4c4f557d6eeb39c11023d668881b9ecfbff48f: Status 404 returned error can't find the container with id 677e4827d4ea856d15e4e4839f4c4f557d6eeb39c11023d668881b9ecfbff48f Jul 29 20:48:03.103497 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:03.103461 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66784977cd-dw2zf" event={"ID":"2f20f2ec-d69d-47cd-8948-0e78ba4c0995","Type":"ContainerStarted","Data":"0cf9499e41ba7224f51105f88b83b0fe221dd284900d85566a0c5622412d24bd"} Jul 29 20:48:03.103989 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:03.103511 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66784977cd-dw2zf" event={"ID":"2f20f2ec-d69d-47cd-8948-0e78ba4c0995","Type":"ContainerStarted","Data":"677e4827d4ea856d15e4e4839f4c4f557d6eeb39c11023d668881b9ecfbff48f"} Jul 29 20:48:03.119904 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:03.119847 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66784977cd-dw2zf" podStartSLOduration=2.119822449 podStartE2EDuration="2.119822449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:48:03.118870349 +0000 UTC m=+99.989973829" watchObservedRunningTime="2026-07-29 20:48:03.119822449 +0000 UTC m=+99.990925918" Jul 29 20:48:08.632850 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:08.632812 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:48:12.211102 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:12.211062 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:12.211505 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:12.211144 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:12.215904 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:12.215883 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:13.154280 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:13.154249 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:48:13.196270 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:13.196236 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57f5bd7f55-2s8kk"] Jul 29 20:48:15.447042 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:15.447002 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:48:15.447428 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:15.447109 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:48:17.235343 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.235302 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" podUID="cc90f71b-6647-4bb2-af21-cfaf42f51393" containerName="registry" containerID="cri-o://054e03aaf95ceed31e5051e97dd08f3112565da3968f89c0ef5568fb3264c560" gracePeriod=30 Jul 29 20:48:17.474842 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.474815 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:48:17.560278 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.560251 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc90f71b-6647-4bb2-af21-cfaf42f51393-ca-trust-extracted\") pod \"cc90f71b-6647-4bb2-af21-cfaf42f51393\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " Jul 29 20:48:17.560439 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.560287 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc90f71b-6647-4bb2-af21-cfaf42f51393-trusted-ca\") pod \"cc90f71b-6647-4bb2-af21-cfaf42f51393\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " Jul 29 20:48:17.560439 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.560356 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-bound-sa-token\") pod \"cc90f71b-6647-4bb2-af21-cfaf42f51393\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " Jul 29 20:48:17.560439 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.560374 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sztj7\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-kube-api-access-sztj7\") pod \"cc90f71b-6647-4bb2-af21-cfaf42f51393\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " Jul 29 20:48:17.560439 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.560392 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc90f71b-6647-4bb2-af21-cfaf42f51393-installation-pull-secrets\") pod \"cc90f71b-6647-4bb2-af21-cfaf42f51393\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " Jul 29 20:48:17.560439 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.560410 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls\") pod \"cc90f71b-6647-4bb2-af21-cfaf42f51393\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " Jul 29 20:48:17.560692 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.560440 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-certificates\") pod \"cc90f71b-6647-4bb2-af21-cfaf42f51393\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " Jul 29 20:48:17.560692 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.560484 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc90f71b-6647-4bb2-af21-cfaf42f51393-image-registry-private-configuration\") pod \"cc90f71b-6647-4bb2-af21-cfaf42f51393\" (UID: \"cc90f71b-6647-4bb2-af21-cfaf42f51393\") " Jul 29 20:48:17.560794 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.560765 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc90f71b-6647-4bb2-af21-cfaf42f51393-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cc90f71b-6647-4bb2-af21-cfaf42f51393" (UID: "cc90f71b-6647-4bb2-af21-cfaf42f51393"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:48:17.560988 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.560937 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cc90f71b-6647-4bb2-af21-cfaf42f51393" (UID: "cc90f71b-6647-4bb2-af21-cfaf42f51393"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:48:17.563190 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.563077 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc90f71b-6647-4bb2-af21-cfaf42f51393-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cc90f71b-6647-4bb2-af21-cfaf42f51393" (UID: "cc90f71b-6647-4bb2-af21-cfaf42f51393"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:48:17.563302 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.563266 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-kube-api-access-sztj7" (OuterVolumeSpecName: "kube-api-access-sztj7") pod "cc90f71b-6647-4bb2-af21-cfaf42f51393" (UID: "cc90f71b-6647-4bb2-af21-cfaf42f51393"). InnerVolumeSpecName "kube-api-access-sztj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:48:17.563353 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.563309 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc90f71b-6647-4bb2-af21-cfaf42f51393-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cc90f71b-6647-4bb2-af21-cfaf42f51393" (UID: "cc90f71b-6647-4bb2-af21-cfaf42f51393"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:48:17.563397 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.563353 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cc90f71b-6647-4bb2-af21-cfaf42f51393" (UID: "cc90f71b-6647-4bb2-af21-cfaf42f51393"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:48:17.563461 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.563438 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cc90f71b-6647-4bb2-af21-cfaf42f51393" (UID: "cc90f71b-6647-4bb2-af21-cfaf42f51393"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:48:17.569295 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.569273 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc90f71b-6647-4bb2-af21-cfaf42f51393-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cc90f71b-6647-4bb2-af21-cfaf42f51393" (UID: "cc90f71b-6647-4bb2-af21-cfaf42f51393"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:48:17.661193 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.661152 2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc90f71b-6647-4bb2-af21-cfaf42f51393-ca-trust-extracted\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:17.661193 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.661185 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc90f71b-6647-4bb2-af21-cfaf42f51393-trusted-ca\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:17.661193 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.661196 2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-bound-sa-token\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:17.661193 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.661205 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sztj7\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-kube-api-access-sztj7\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:17.661456 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.661214 2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc90f71b-6647-4bb2-af21-cfaf42f51393-installation-pull-secrets\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:17.661456 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.661223 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-tls\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:17.661456 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.661231 2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc90f71b-6647-4bb2-af21-cfaf42f51393-registry-certificates\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:17.661456 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:17.661240 2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc90f71b-6647-4bb2-af21-cfaf42f51393-image-registry-private-configuration\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:18.173018 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:18.172983 2562 generic.go:356] "Generic (PLEG): container finished" podID="cc90f71b-6647-4bb2-af21-cfaf42f51393" containerID="054e03aaf95ceed31e5051e97dd08f3112565da3968f89c0ef5568fb3264c560" exitCode=0 Jul 29 20:48:18.173225 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:18.173028 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" Jul 29 20:48:18.173225 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:18.173077 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" event={"ID":"cc90f71b-6647-4bb2-af21-cfaf42f51393","Type":"ContainerDied","Data":"054e03aaf95ceed31e5051e97dd08f3112565da3968f89c0ef5568fb3264c560"} Jul 29 20:48:18.173225 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:18.173134 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f68896cdc-qpf7g" event={"ID":"cc90f71b-6647-4bb2-af21-cfaf42f51393","Type":"ContainerDied","Data":"2682efc264448889ea663205b3aa0160b2274b993545b33d44c9dab9e0b3fd97"} Jul 29 20:48:18.173225 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:18.173163 2562 scope.go:117] "RemoveContainer" containerID="054e03aaf95ceed31e5051e97dd08f3112565da3968f89c0ef5568fb3264c560" Jul 29 20:48:18.184470 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:18.184450 2562 scope.go:117] "RemoveContainer" containerID="054e03aaf95ceed31e5051e97dd08f3112565da3968f89c0ef5568fb3264c560" Jul 29 20:48:18.184787 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:48:18.184764 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"054e03aaf95ceed31e5051e97dd08f3112565da3968f89c0ef5568fb3264c560\": container with ID starting with 054e03aaf95ceed31e5051e97dd08f3112565da3968f89c0ef5568fb3264c560 not found: ID does not exist" containerID="054e03aaf95ceed31e5051e97dd08f3112565da3968f89c0ef5568fb3264c560" Jul 29 20:48:18.184843 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:18.184794 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"054e03aaf95ceed31e5051e97dd08f3112565da3968f89c0ef5568fb3264c560"} err="failed to get container status \"054e03aaf95ceed31e5051e97dd08f3112565da3968f89c0ef5568fb3264c560\": rpc error: code = NotFound desc = could not find container \"054e03aaf95ceed31e5051e97dd08f3112565da3968f89c0ef5568fb3264c560\": container with ID starting with 054e03aaf95ceed31e5051e97dd08f3112565da3968f89c0ef5568fb3264c560 not found: ID does not exist" Jul 29 20:48:18.192487 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:18.192463 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f68896cdc-qpf7g"] Jul 29 20:48:18.196090 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:18.196069 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f68896cdc-qpf7g"] Jul 29 20:48:19.556011 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:19.555979 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc90f71b-6647-4bb2-af21-cfaf42f51393" path="/var/lib/kubelet/pods/cc90f71b-6647-4bb2-af21-cfaf42f51393/volumes" Jul 29 20:48:24.201624 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:24.201576 2562 generic.go:356] "Generic (PLEG): container finished" podID="a08729b6-2792-466e-b7c4-fb0ae44087e6" containerID="16d3e61abc4a0defa0096fd045f572c80a8d3e112c00aae109ca52e5d53ed53e" exitCode=0 Jul 29 20:48:24.202035 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:24.201655 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-9hbmn" event={"ID":"a08729b6-2792-466e-b7c4-fb0ae44087e6","Type":"ContainerDied","Data":"16d3e61abc4a0defa0096fd045f572c80a8d3e112c00aae109ca52e5d53ed53e"} Jul 29 20:48:24.202090 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:24.202070 2562 scope.go:117] "RemoveContainer" containerID="16d3e61abc4a0defa0096fd045f572c80a8d3e112c00aae109ca52e5d53ed53e" Jul 29 20:48:24.852906 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:24.852877 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f46dc85cf-9562c_d56bff06-5480-4bcc-bf5b-dbfd2372f9cd/router/0.log" Jul 29 20:48:24.868089 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:24.868061 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zwgsv_43c0ba7b-1895-4952-b45b-87f2983002d1/serve-healthcheck-canary/0.log" Jul 29 20:48:25.207811 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:25.207724 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-9hbmn" event={"ID":"a08729b6-2792-466e-b7c4-fb0ae44087e6","Type":"ContainerStarted","Data":"d99bceaa1a7c846990325b26bee62d09f42b26e70b2175df5f4442cf6983a80f"} Jul 29 20:48:26.845983 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:26.845932 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bb6b5b74f-wrrwg" podUID="fefe1fd2-a131-4c08-bbc6-2d634e86467c" containerName="console" containerID="cri-o://e89bcadd1b0b045cb0a66617c85bb36dedd500a0ca8cab8eb6b6d869799bc81e" gracePeriod=15 Jul 29 20:48:27.092253 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.092223 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:48:27.141791 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.141689 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fefe1fd2-a131-4c08-bbc6-2d634e86467c-console-serving-cert\") pod \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " Jul 29 20:48:27.141791 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.141728 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fefe1fd2-a131-4c08-bbc6-2d634e86467c-console-oauth-config\") pod \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " Jul 29 20:48:27.141791 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.141749 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-service-ca\") pod \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " Jul 29 20:48:27.141791 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.141774 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-trusted-ca-bundle\") pod \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " Jul 29 20:48:27.142151 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.141798 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58sv8\" (UniqueName: \"kubernetes.io/projected/fefe1fd2-a131-4c08-bbc6-2d634e86467c-kube-api-access-58sv8\") pod \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " Jul 29 20:48:27.142151 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.141939 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-console-config\") pod \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " Jul 29 20:48:27.142151 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.141964 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-oauth-serving-cert\") pod \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\" (UID: \"fefe1fd2-a131-4c08-bbc6-2d634e86467c\") " Jul 29 20:48:27.142323 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.142169 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-service-ca" (OuterVolumeSpecName: "service-ca") pod "fefe1fd2-a131-4c08-bbc6-2d634e86467c" (UID: "fefe1fd2-a131-4c08-bbc6-2d634e86467c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:48:27.142323 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.142210 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fefe1fd2-a131-4c08-bbc6-2d634e86467c" (UID: "fefe1fd2-a131-4c08-bbc6-2d634e86467c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:48:27.142494 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.142467 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fefe1fd2-a131-4c08-bbc6-2d634e86467c" (UID: "fefe1fd2-a131-4c08-bbc6-2d634e86467c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:48:27.142494 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.142492 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-console-config" (OuterVolumeSpecName: "console-config") pod "fefe1fd2-a131-4c08-bbc6-2d634e86467c" (UID: "fefe1fd2-a131-4c08-bbc6-2d634e86467c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:48:27.144083 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.144056 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fefe1fd2-a131-4c08-bbc6-2d634e86467c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fefe1fd2-a131-4c08-bbc6-2d634e86467c" (UID: "fefe1fd2-a131-4c08-bbc6-2d634e86467c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:48:27.144320 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.144297 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fefe1fd2-a131-4c08-bbc6-2d634e86467c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fefe1fd2-a131-4c08-bbc6-2d634e86467c" (UID: "fefe1fd2-a131-4c08-bbc6-2d634e86467c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:48:27.144393 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.144344 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefe1fd2-a131-4c08-bbc6-2d634e86467c-kube-api-access-58sv8" (OuterVolumeSpecName: "kube-api-access-58sv8") pod "fefe1fd2-a131-4c08-bbc6-2d634e86467c" (UID: "fefe1fd2-a131-4c08-bbc6-2d634e86467c"). InnerVolumeSpecName "kube-api-access-58sv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:48:27.216971 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.216904 2562 generic.go:356] "Generic (PLEG): container finished" podID="fefe1fd2-a131-4c08-bbc6-2d634e86467c" containerID="e89bcadd1b0b045cb0a66617c85bb36dedd500a0ca8cab8eb6b6d869799bc81e" exitCode=0 Jul 29 20:48:27.217215 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.216979 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bb6b5b74f-wrrwg" Jul 29 20:48:27.217215 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.216996 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bb6b5b74f-wrrwg" event={"ID":"fefe1fd2-a131-4c08-bbc6-2d634e86467c","Type":"ContainerDied","Data":"e89bcadd1b0b045cb0a66617c85bb36dedd500a0ca8cab8eb6b6d869799bc81e"} Jul 29 20:48:27.217215 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.217060 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bb6b5b74f-wrrwg" event={"ID":"fefe1fd2-a131-4c08-bbc6-2d634e86467c","Type":"ContainerDied","Data":"3a419e437e01a78b49a8edaaa8df6856a877d5fcb8de886999c7f7afcb5a1c4c"} Jul 29 20:48:27.217215 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.217083 2562 scope.go:117] "RemoveContainer" containerID="e89bcadd1b0b045cb0a66617c85bb36dedd500a0ca8cab8eb6b6d869799bc81e" Jul 29 20:48:27.225792 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.225775 2562 scope.go:117] "RemoveContainer" containerID="e89bcadd1b0b045cb0a66617c85bb36dedd500a0ca8cab8eb6b6d869799bc81e" Jul 29 20:48:27.226085 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:48:27.226065 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e89bcadd1b0b045cb0a66617c85bb36dedd500a0ca8cab8eb6b6d869799bc81e\": container with ID starting with e89bcadd1b0b045cb0a66617c85bb36dedd500a0ca8cab8eb6b6d869799bc81e not found: ID does not exist" containerID="e89bcadd1b0b045cb0a66617c85bb36dedd500a0ca8cab8eb6b6d869799bc81e" Jul 29 20:48:27.226165 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.226091 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89bcadd1b0b045cb0a66617c85bb36dedd500a0ca8cab8eb6b6d869799bc81e"} err="failed to get container status \"e89bcadd1b0b045cb0a66617c85bb36dedd500a0ca8cab8eb6b6d869799bc81e\": rpc error: code = NotFound desc = could not find container \"e89bcadd1b0b045cb0a66617c85bb36dedd500a0ca8cab8eb6b6d869799bc81e\": container with ID starting with e89bcadd1b0b045cb0a66617c85bb36dedd500a0ca8cab8eb6b6d869799bc81e not found: ID does not exist" Jul 29 20:48:27.236663 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.236636 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bb6b5b74f-wrrwg"] Jul 29 20:48:27.239598 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.239575 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bb6b5b74f-wrrwg"] Jul 29 20:48:27.243396 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.243379 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-console-config\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:27.243478 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.243397 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-oauth-serving-cert\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:27.243478 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.243407 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fefe1fd2-a131-4c08-bbc6-2d634e86467c-console-serving-cert\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:27.243478 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.243417 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fefe1fd2-a131-4c08-bbc6-2d634e86467c-console-oauth-config\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:27.243478 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.243426 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-service-ca\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:27.243478 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.243434 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fefe1fd2-a131-4c08-bbc6-2d634e86467c-trusted-ca-bundle\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:27.243478 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.243443 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58sv8\" (UniqueName: \"kubernetes.io/projected/fefe1fd2-a131-4c08-bbc6-2d634e86467c-kube-api-access-58sv8\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:27.556804 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:27.556773 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fefe1fd2-a131-4c08-bbc6-2d634e86467c" path="/var/lib/kubelet/pods/fefe1fd2-a131-4c08-bbc6-2d634e86467c/volumes" Jul 29 20:48:30.230647 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:30.230611 2562 generic.go:356] "Generic (PLEG): container finished" podID="4250508f-31ea-4575-b84e-50a57d1aa159" containerID="2b8a78557610d2d3c452c2089e07939d73d54f14fef2dffd52a7226e6f09cab7" exitCode=0 Jul 29 20:48:30.231054 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:30.230686 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7" event={"ID":"4250508f-31ea-4575-b84e-50a57d1aa159","Type":"ContainerDied","Data":"2b8a78557610d2d3c452c2089e07939d73d54f14fef2dffd52a7226e6f09cab7"} Jul 29 20:48:30.231103 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:30.231055 2562 scope.go:117] "RemoveContainer" containerID="2b8a78557610d2d3c452c2089e07939d73d54f14fef2dffd52a7226e6f09cab7" Jul 29 20:48:31.235855 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:31.235819 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7" event={"ID":"4250508f-31ea-4575-b84e-50a57d1aa159","Type":"ContainerStarted","Data":"6064c0e7790d9bbd9f6fa1e346b0e6b8111e2d90c2c44133224aa2d690fb5e4e"} Jul 29 20:48:35.452455 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:35.452418 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:48:35.456400 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:35.456374 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-577c88c6f7-qb7xj" Jul 29 20:48:38.219060 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.219022 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57f5bd7f55-2s8kk" podUID="7119b239-5157-44f0-aecc-1fef1d56a20b" containerName="console" containerID="cri-o://35929220d890ccae5f1b84f33b3c1b65a3b84cffed9cc8753f144efc8c6b40f9" gracePeriod=15 Jul 29 20:48:38.486000 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.485910 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:48:38.538570 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.538527 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7119b239-5157-44f0-aecc-1fef1d56a20b-console-oauth-config\") pod \"7119b239-5157-44f0-aecc-1fef1d56a20b\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " Jul 29 20:48:38.538749 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.538589 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7119b239-5157-44f0-aecc-1fef1d56a20b-console-config\") pod \"7119b239-5157-44f0-aecc-1fef1d56a20b\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " Jul 29 20:48:38.538749 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.538619 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7119b239-5157-44f0-aecc-1fef1d56a20b-console-serving-cert\") pod \"7119b239-5157-44f0-aecc-1fef1d56a20b\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " Jul 29 20:48:38.538749 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.538664 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4px9h\" (UniqueName: \"kubernetes.io/projected/7119b239-5157-44f0-aecc-1fef1d56a20b-kube-api-access-4px9h\") pod \"7119b239-5157-44f0-aecc-1fef1d56a20b\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " Jul 29 20:48:38.538749 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.538711 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7119b239-5157-44f0-aecc-1fef1d56a20b-oauth-serving-cert\") pod \"7119b239-5157-44f0-aecc-1fef1d56a20b\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " Jul 29 20:48:38.539026 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.538755 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7119b239-5157-44f0-aecc-1fef1d56a20b-service-ca\") pod \"7119b239-5157-44f0-aecc-1fef1d56a20b\" (UID: \"7119b239-5157-44f0-aecc-1fef1d56a20b\") " Jul 29 20:48:38.539090 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.539050 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7119b239-5157-44f0-aecc-1fef1d56a20b-console-config" (OuterVolumeSpecName: "console-config") pod "7119b239-5157-44f0-aecc-1fef1d56a20b" (UID: "7119b239-5157-44f0-aecc-1fef1d56a20b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:48:38.539299 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.539268 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7119b239-5157-44f0-aecc-1fef1d56a20b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7119b239-5157-44f0-aecc-1fef1d56a20b" (UID: "7119b239-5157-44f0-aecc-1fef1d56a20b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:48:38.539400 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.539328 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7119b239-5157-44f0-aecc-1fef1d56a20b-service-ca" (OuterVolumeSpecName: "service-ca") pod "7119b239-5157-44f0-aecc-1fef1d56a20b" (UID: "7119b239-5157-44f0-aecc-1fef1d56a20b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:48:38.541342 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.541310 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7119b239-5157-44f0-aecc-1fef1d56a20b-kube-api-access-4px9h" (OuterVolumeSpecName: "kube-api-access-4px9h") pod "7119b239-5157-44f0-aecc-1fef1d56a20b" (UID: "7119b239-5157-44f0-aecc-1fef1d56a20b"). InnerVolumeSpecName "kube-api-access-4px9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:48:38.541426 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.541343 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7119b239-5157-44f0-aecc-1fef1d56a20b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7119b239-5157-44f0-aecc-1fef1d56a20b" (UID: "7119b239-5157-44f0-aecc-1fef1d56a20b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:48:38.541537 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.541515 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7119b239-5157-44f0-aecc-1fef1d56a20b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7119b239-5157-44f0-aecc-1fef1d56a20b" (UID: "7119b239-5157-44f0-aecc-1fef1d56a20b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:48:38.639680 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.639648 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7119b239-5157-44f0-aecc-1fef1d56a20b-console-oauth-config\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:38.639680 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.639678 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7119b239-5157-44f0-aecc-1fef1d56a20b-console-config\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:38.639873 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.639693 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7119b239-5157-44f0-aecc-1fef1d56a20b-console-serving-cert\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:38.639873 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.639706 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4px9h\" (UniqueName: \"kubernetes.io/projected/7119b239-5157-44f0-aecc-1fef1d56a20b-kube-api-access-4px9h\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:38.639873 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.639717 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7119b239-5157-44f0-aecc-1fef1d56a20b-oauth-serving-cert\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:38.639873 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:38.639730 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7119b239-5157-44f0-aecc-1fef1d56a20b-service-ca\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:48:39.274130 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:39.274033 2562 generic.go:356] "Generic (PLEG): container finished" podID="863cac4c-638f-4789-88d9-263045a9c8ac" containerID="c3362fc539261f9fa82f21590c69ea1956e9d4c70a60be5b4ec26b89cb88cf10" exitCode=0 Jul 29 20:48:39.274558 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:39.274172 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vmz8d" event={"ID":"863cac4c-638f-4789-88d9-263045a9c8ac","Type":"ContainerDied","Data":"c3362fc539261f9fa82f21590c69ea1956e9d4c70a60be5b4ec26b89cb88cf10"} Jul 29 20:48:39.274645 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:39.274618 2562 scope.go:117] "RemoveContainer" containerID="c3362fc539261f9fa82f21590c69ea1956e9d4c70a60be5b4ec26b89cb88cf10" Jul 29 20:48:39.276912 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:39.276892 2562 generic.go:356] "Generic (PLEG): container finished" podID="7119b239-5157-44f0-aecc-1fef1d56a20b" containerID="35929220d890ccae5f1b84f33b3c1b65a3b84cffed9cc8753f144efc8c6b40f9" exitCode=0 Jul 29 20:48:39.277029 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:39.276954 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f5bd7f55-2s8kk" event={"ID":"7119b239-5157-44f0-aecc-1fef1d56a20b","Type":"ContainerDied","Data":"35929220d890ccae5f1b84f33b3c1b65a3b84cffed9cc8753f144efc8c6b40f9"} Jul 29 20:48:39.277029 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:39.277007 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f5bd7f55-2s8kk" event={"ID":"7119b239-5157-44f0-aecc-1fef1d56a20b","Type":"ContainerDied","Data":"a024804ba4ef4eaa07b8470d683ac7e33510894875e1af791d20bf1bc8f1546c"} Jul 29 20:48:39.277126 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:39.277032 2562 scope.go:117] "RemoveContainer" containerID="35929220d890ccae5f1b84f33b3c1b65a3b84cffed9cc8753f144efc8c6b40f9" Jul 29 20:48:39.277126 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:39.276967 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f5bd7f55-2s8kk" Jul 29 20:48:39.286444 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:39.286353 2562 scope.go:117] "RemoveContainer" containerID="35929220d890ccae5f1b84f33b3c1b65a3b84cffed9cc8753f144efc8c6b40f9" Jul 29 20:48:39.286713 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:48:39.286687 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35929220d890ccae5f1b84f33b3c1b65a3b84cffed9cc8753f144efc8c6b40f9\": container with ID starting with 35929220d890ccae5f1b84f33b3c1b65a3b84cffed9cc8753f144efc8c6b40f9 not found: ID does not exist" containerID="35929220d890ccae5f1b84f33b3c1b65a3b84cffed9cc8753f144efc8c6b40f9" Jul 29 20:48:39.286788 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:39.286722 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35929220d890ccae5f1b84f33b3c1b65a3b84cffed9cc8753f144efc8c6b40f9"} err="failed to get container status \"35929220d890ccae5f1b84f33b3c1b65a3b84cffed9cc8753f144efc8c6b40f9\": rpc error: code = NotFound desc = could not find container \"35929220d890ccae5f1b84f33b3c1b65a3b84cffed9cc8753f144efc8c6b40f9\": container with ID starting with 35929220d890ccae5f1b84f33b3c1b65a3b84cffed9cc8753f144efc8c6b40f9 not found: ID does not exist" Jul 29 20:48:39.307057 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:39.307026 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57f5bd7f55-2s8kk"] Jul 29 20:48:39.317217 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:39.317189 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57f5bd7f55-2s8kk"] Jul 29 20:48:39.555930 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:39.555892 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7119b239-5157-44f0-aecc-1fef1d56a20b" path="/var/lib/kubelet/pods/7119b239-5157-44f0-aecc-1fef1d56a20b/volumes" Jul 29 20:48:40.282821 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:48:40.282789 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vmz8d" event={"ID":"863cac4c-638f-4789-88d9-263045a9c8ac","Type":"ContainerStarted","Data":"020bcfd90ce9afb47a04f67167312af583557021746c66c742eef05f438e024c"} Jul 29 20:49:09.251046 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.251012 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bc6c7f784-bhnlb"] Jul 29 20:49:09.251497 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.251346 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc90f71b-6647-4bb2-af21-cfaf42f51393" containerName="registry" Jul 29 20:49:09.251497 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.251357 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc90f71b-6647-4bb2-af21-cfaf42f51393" containerName="registry" Jul 29 20:49:09.251497 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.251372 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7119b239-5157-44f0-aecc-1fef1d56a20b" containerName="console" Jul 29 20:49:09.251497 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.251377 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7119b239-5157-44f0-aecc-1fef1d56a20b" containerName="console" Jul 29 20:49:09.251497 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.251389 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fefe1fd2-a131-4c08-bbc6-2d634e86467c" containerName="console" Jul 29 20:49:09.251497 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.251394 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fefe1fd2-a131-4c08-bbc6-2d634e86467c" containerName="console" Jul 29 20:49:09.251497 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.251452 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc90f71b-6647-4bb2-af21-cfaf42f51393" containerName="registry" Jul 29 20:49:09.251497 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.251460 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fefe1fd2-a131-4c08-bbc6-2d634e86467c" containerName="console" Jul 29 20:49:09.251497 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.251466 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7119b239-5157-44f0-aecc-1fef1d56a20b" containerName="console" Jul 29 20:49:09.255963 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.255937 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.267349 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.267323 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bc6c7f784-bhnlb"] Jul 29 20:49:09.407453 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.407410 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-console-config\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.407637 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.407492 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-oauth-serving-cert\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.407637 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.407521 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5009db96-8df4-4785-858a-0c2ab892c94a-console-serving-cert\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.407637 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.407562 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-trusted-ca-bundle\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.407637 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.407628 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-service-ca\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.407771 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.407656 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5009db96-8df4-4785-858a-0c2ab892c94a-console-oauth-config\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.407771 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.407680 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjz8w\" (UniqueName: \"kubernetes.io/projected/5009db96-8df4-4785-858a-0c2ab892c94a-kube-api-access-sjz8w\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.508580 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.508498 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-oauth-serving-cert\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.508580 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.508531 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5009db96-8df4-4785-858a-0c2ab892c94a-console-serving-cert\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.508580 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.508563 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-trusted-ca-bundle\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.508814 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.508601 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-service-ca\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.508814 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.508626 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5009db96-8df4-4785-858a-0c2ab892c94a-console-oauth-config\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.508814 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.508648 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjz8w\" (UniqueName: \"kubernetes.io/projected/5009db96-8df4-4785-858a-0c2ab892c94a-kube-api-access-sjz8w\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.508814 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.508676 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-console-config\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.509367 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.509342 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-console-config\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.509491 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.509375 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-oauth-serving-cert\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.509491 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.509471 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-trusted-ca-bundle\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.509828 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.509808 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-service-ca\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.511067 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.511039 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5009db96-8df4-4785-858a-0c2ab892c94a-console-serving-cert\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.511162 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.511106 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5009db96-8df4-4785-858a-0c2ab892c94a-console-oauth-config\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.515861 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.515844 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjz8w\" (UniqueName: \"kubernetes.io/projected/5009db96-8df4-4785-858a-0c2ab892c94a-kube-api-access-sjz8w\") pod \"console-bc6c7f784-bhnlb\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.565503 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.565466 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:09.695992 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:09.695968 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bc6c7f784-bhnlb"] Jul 29 20:49:09.698040 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:49:09.698012 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5009db96_8df4_4785_858a_0c2ab892c94a.slice/crio-c925fa1e0f720b21e095eeedea0dbf259215db9f1d67df529637f0e60478a1c8 WatchSource:0}: Error finding container c925fa1e0f720b21e095eeedea0dbf259215db9f1d67df529637f0e60478a1c8: Status 404 returned error can't find the container with id c925fa1e0f720b21e095eeedea0dbf259215db9f1d67df529637f0e60478a1c8 Jul 29 20:49:10.404202 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:10.404166 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bc6c7f784-bhnlb" event={"ID":"5009db96-8df4-4785-858a-0c2ab892c94a","Type":"ContainerStarted","Data":"80adcf83691aad1229f47ed364721cf9fd92c059acf443a21057aed7e1a872be"} Jul 29 20:49:10.404664 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:10.404226 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bc6c7f784-bhnlb" event={"ID":"5009db96-8df4-4785-858a-0c2ab892c94a","Type":"ContainerStarted","Data":"c925fa1e0f720b21e095eeedea0dbf259215db9f1d67df529637f0e60478a1c8"} Jul 29 20:49:10.420855 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:10.420796 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bc6c7f784-bhnlb" podStartSLOduration=1.420780464 podStartE2EDuration="1.420780464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:49:10.418902944 +0000 UTC m=+167.290006413" watchObservedRunningTime="2026-07-29 20:49:10.420780464 +0000 UTC m=+167.291883994" Jul 29 20:49:11.161080 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:11.161043 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 20:49:11.161342 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:11.161319 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="alertmanager" containerID="cri-o://cc79d8eb057249931dc8707c73aa3faad29d2d14b1e46301ef658c09bef1072a" gracePeriod=120 Jul 29 20:49:11.161426 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:11.161390 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="kube-rbac-proxy-metric" containerID="cri-o://0b737f2f659b6aa832474270e109c6dcf82980e5d43cd9c0f5dfab63e5d3db6b" gracePeriod=120 Jul 29 20:49:11.161515 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:11.161403 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="kube-rbac-proxy-web" containerID="cri-o://84e7f355a92d174b8b898653580e9551bc8f788e52f6e70c234f70d814fd374d" gracePeriod=120 Jul 29 20:49:11.161515 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:11.161437 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="config-reloader" containerID="cri-o://8327c154ba2f4110332b0a54274de9aa3ea4c64fbc11bd07b00a4ebed5b58941" gracePeriod=120 Jul 29 20:49:11.161515 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:11.161487 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="prom-label-proxy" containerID="cri-o://3b40e92336cf7b8b5104dec082302f6b08466ead39b6b9998e9249dbe0312ff9" gracePeriod=120 Jul 29 20:49:11.161665 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:11.161409 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="kube-rbac-proxy" containerID="cri-o://a00cb10010c66e4afc640e30b06abff6d2c8986c049fdda13fcacf6f48540480" gracePeriod=120 Jul 29 20:49:11.411322 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:11.411234 2562 generic.go:356] "Generic (PLEG): container finished" podID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerID="3b40e92336cf7b8b5104dec082302f6b08466ead39b6b9998e9249dbe0312ff9" exitCode=0 Jul 29 20:49:11.411322 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:11.411260 2562 generic.go:356] "Generic (PLEG): container finished" podID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerID="a00cb10010c66e4afc640e30b06abff6d2c8986c049fdda13fcacf6f48540480" exitCode=0 Jul 29 20:49:11.411322 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:11.411267 2562 generic.go:356] "Generic (PLEG): container finished" podID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerID="8327c154ba2f4110332b0a54274de9aa3ea4c64fbc11bd07b00a4ebed5b58941" exitCode=0 Jul 29 20:49:11.411322 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:11.411272 2562 generic.go:356] "Generic (PLEG): container finished" podID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerID="cc79d8eb057249931dc8707c73aa3faad29d2d14b1e46301ef658c09bef1072a" exitCode=0 Jul 29 20:49:11.411322 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:11.411308 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01ec765f-ea51-44ac-b38c-5eb565d63f6f","Type":"ContainerDied","Data":"3b40e92336cf7b8b5104dec082302f6b08466ead39b6b9998e9249dbe0312ff9"} Jul 29 20:49:11.411800 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:11.411375 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01ec765f-ea51-44ac-b38c-5eb565d63f6f","Type":"ContainerDied","Data":"a00cb10010c66e4afc640e30b06abff6d2c8986c049fdda13fcacf6f48540480"} Jul 29 20:49:11.411800 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:11.411403 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01ec765f-ea51-44ac-b38c-5eb565d63f6f","Type":"ContainerDied","Data":"8327c154ba2f4110332b0a54274de9aa3ea4c64fbc11bd07b00a4ebed5b58941"} Jul 29 20:49:11.411800 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:11.411427 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01ec765f-ea51-44ac-b38c-5eb565d63f6f","Type":"ContainerDied","Data":"cc79d8eb057249931dc8707c73aa3faad29d2d14b1e46301ef658c09bef1072a"} Jul 29 20:49:12.419032 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.419002 2562 generic.go:356] "Generic (PLEG): container finished" podID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerID="0b737f2f659b6aa832474270e109c6dcf82980e5d43cd9c0f5dfab63e5d3db6b" exitCode=0 Jul 29 20:49:12.419032 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.419027 2562 generic.go:356] "Generic (PLEG): container finished" podID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerID="84e7f355a92d174b8b898653580e9551bc8f788e52f6e70c234f70d814fd374d" exitCode=0 Jul 29 20:49:12.419405 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.419064 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01ec765f-ea51-44ac-b38c-5eb565d63f6f","Type":"ContainerDied","Data":"0b737f2f659b6aa832474270e109c6dcf82980e5d43cd9c0f5dfab63e5d3db6b"} Jul 29 20:49:12.419405 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.419119 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01ec765f-ea51-44ac-b38c-5eb565d63f6f","Type":"ContainerDied","Data":"84e7f355a92d174b8b898653580e9551bc8f788e52f6e70c234f70d814fd374d"} Jul 29 20:49:12.419405 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.419143 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01ec765f-ea51-44ac-b38c-5eb565d63f6f","Type":"ContainerDied","Data":"143548c4be64977b6d791599d2cb5727648428ce915293995b8f4fab3c9229c2"} Jul 29 20:49:12.419405 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.419167 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="143548c4be64977b6d791599d2cb5727648428ce915293995b8f4fab3c9229c2" Jul 29 20:49:12.419572 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.419402 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:12.535680 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.535589 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ec765f-ea51-44ac-b38c-5eb565d63f6f-alertmanager-trusted-ca-bundle\") pod \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " Jul 29 20:49:12.535680 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.535649 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/01ec765f-ea51-44ac-b38c-5eb565d63f6f-alertmanager-main-db\") pod \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " Jul 29 20:49:12.535680 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.535674 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-kube-rbac-proxy\") pod \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " Jul 29 20:49:12.535902 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.535833 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-cluster-tls-config\") pod \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " Jul 29 20:49:12.535902 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.535864 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-config-volume\") pod \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " Jul 29 20:49:12.535902 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.535896 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " Jul 29 20:49:12.536108 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.535954 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-kube-rbac-proxy-web\") pod \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " Jul 29 20:49:12.536108 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.535985 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01ec765f-ea51-44ac-b38c-5eb565d63f6f-config-out\") pod \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " Jul 29 20:49:12.536108 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.536006 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ec765f-ea51-44ac-b38c-5eb565d63f6f-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "01ec765f-ea51-44ac-b38c-5eb565d63f6f" (UID: "01ec765f-ea51-44ac-b38c-5eb565d63f6f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:49:12.536108 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.536039 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-web-config\") pod \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " Jul 29 20:49:12.536108 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.536040 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ec765f-ea51-44ac-b38c-5eb565d63f6f-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "01ec765f-ea51-44ac-b38c-5eb565d63f6f" (UID: "01ec765f-ea51-44ac-b38c-5eb565d63f6f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:49:12.536108 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.536099 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01ec765f-ea51-44ac-b38c-5eb565d63f6f-tls-assets\") pod \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " Jul 29 20:49:12.536433 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.536146 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k9lj\" (UniqueName: \"kubernetes.io/projected/01ec765f-ea51-44ac-b38c-5eb565d63f6f-kube-api-access-2k9lj\") pod \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " Jul 29 20:49:12.536433 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.536170 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-main-tls\") pod \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " Jul 29 20:49:12.536433 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.536198 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01ec765f-ea51-44ac-b38c-5eb565d63f6f-metrics-client-ca\") pod \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\" (UID: \"01ec765f-ea51-44ac-b38c-5eb565d63f6f\") " Jul 29 20:49:12.536616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.536452 2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/01ec765f-ea51-44ac-b38c-5eb565d63f6f-alertmanager-main-db\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:12.536616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.536472 2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01ec765f-ea51-44ac-b38c-5eb565d63f6f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:12.536842 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.536815 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ec765f-ea51-44ac-b38c-5eb565d63f6f-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "01ec765f-ea51-44ac-b38c-5eb565d63f6f" (UID: "01ec765f-ea51-44ac-b38c-5eb565d63f6f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:49:12.538938 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.538885 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "01ec765f-ea51-44ac-b38c-5eb565d63f6f" (UID: "01ec765f-ea51-44ac-b38c-5eb565d63f6f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:49:12.539278 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.539227 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ec765f-ea51-44ac-b38c-5eb565d63f6f-config-out" (OuterVolumeSpecName: "config-out") pod "01ec765f-ea51-44ac-b38c-5eb565d63f6f" (UID: "01ec765f-ea51-44ac-b38c-5eb565d63f6f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:49:12.539278 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.539234 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-config-volume" (OuterVolumeSpecName: "config-volume") pod "01ec765f-ea51-44ac-b38c-5eb565d63f6f" (UID: "01ec765f-ea51-44ac-b38c-5eb565d63f6f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:49:12.539457 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.539288 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "01ec765f-ea51-44ac-b38c-5eb565d63f6f" (UID: "01ec765f-ea51-44ac-b38c-5eb565d63f6f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:49:12.539457 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.539314 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "01ec765f-ea51-44ac-b38c-5eb565d63f6f" (UID: "01ec765f-ea51-44ac-b38c-5eb565d63f6f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:49:12.539457 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.539330 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ec765f-ea51-44ac-b38c-5eb565d63f6f-kube-api-access-2k9lj" (OuterVolumeSpecName: "kube-api-access-2k9lj") pod "01ec765f-ea51-44ac-b38c-5eb565d63f6f" (UID: "01ec765f-ea51-44ac-b38c-5eb565d63f6f"). InnerVolumeSpecName "kube-api-access-2k9lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:49:12.539700 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.539679 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "01ec765f-ea51-44ac-b38c-5eb565d63f6f" (UID: "01ec765f-ea51-44ac-b38c-5eb565d63f6f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:49:12.540202 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.540180 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ec765f-ea51-44ac-b38c-5eb565d63f6f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "01ec765f-ea51-44ac-b38c-5eb565d63f6f" (UID: "01ec765f-ea51-44ac-b38c-5eb565d63f6f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:49:12.543045 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.543019 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "01ec765f-ea51-44ac-b38c-5eb565d63f6f" (UID: "01ec765f-ea51-44ac-b38c-5eb565d63f6f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:49:12.549908 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.549881 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-web-config" (OuterVolumeSpecName: "web-config") pod "01ec765f-ea51-44ac-b38c-5eb565d63f6f" (UID: "01ec765f-ea51-44ac-b38c-5eb565d63f6f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:49:12.637801 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.637752 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:12.637801 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.637798 2562 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-cluster-tls-config\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:12.637801 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.637809 2562 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-config-volume\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:12.638071 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.637821 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:12.638071 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.637830 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:12.638071 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.637839 2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01ec765f-ea51-44ac-b38c-5eb565d63f6f-config-out\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:12.638071 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.637848 2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-web-config\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:12.638071 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.637855 2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01ec765f-ea51-44ac-b38c-5eb565d63f6f-tls-assets\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:12.638071 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.637863 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k9lj\" (UniqueName: \"kubernetes.io/projected/01ec765f-ea51-44ac-b38c-5eb565d63f6f-kube-api-access-2k9lj\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:12.638071 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.637873 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/01ec765f-ea51-44ac-b38c-5eb565d63f6f-secret-alertmanager-main-tls\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:12.638071 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:12.637882 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01ec765f-ea51-44ac-b38c-5eb565d63f6f-metrics-client-ca\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:13.423089 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.423049 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.444849 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.444818 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 20:49:13.450144 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.450121 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 20:49:13.473215 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473154 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 20:49:13.473478 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473466 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="config-reloader" Jul 29 20:49:13.473521 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473480 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="config-reloader" Jul 29 20:49:13.473521 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473495 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="prom-label-proxy" Jul 29 20:49:13.473521 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473501 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="prom-label-proxy" Jul 29 20:49:13.473521 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473511 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="alertmanager" Jul 29 20:49:13.473521 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473517 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="alertmanager" Jul 29 20:49:13.473703 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473526 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="kube-rbac-proxy-web" Jul 29 20:49:13.473703 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473531 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="kube-rbac-proxy-web" Jul 29 20:49:13.473703 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473538 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="kube-rbac-proxy-metric" Jul 29 20:49:13.473703 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473544 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="kube-rbac-proxy-metric" Jul 29 20:49:13.473703 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473550 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="init-config-reloader" Jul 29 20:49:13.473703 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473555 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="init-config-reloader" Jul 29 20:49:13.473703 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473566 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="kube-rbac-proxy" Jul 29 20:49:13.473703 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473571 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="kube-rbac-proxy" Jul 29 20:49:13.473703 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473615 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="kube-rbac-proxy-metric" Jul 29 20:49:13.473703 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473622 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="alertmanager" Jul 29 20:49:13.473703 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473629 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="kube-rbac-proxy" Jul 29 20:49:13.473703 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473636 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="prom-label-proxy" Jul 29 20:49:13.473703 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473642 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="kube-rbac-proxy-web" Jul 29 20:49:13.473703 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.473649 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" containerName="config-reloader" Jul 29 20:49:13.478287 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.478268 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.480187 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.480167 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 29 20:49:13.480534 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.480508 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 29 20:49:13.480607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.480514 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 29 20:49:13.480670 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.480534 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 29 20:49:13.480670 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.480534 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 29 20:49:13.480843 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.480823 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 29 20:49:13.480946 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.480906 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wzfqg\"" Jul 29 20:49:13.481001 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.480958 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 29 20:49:13.481001 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.480976 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 29 20:49:13.486230 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.486214 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 29 20:49:13.489948 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.489909 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 20:49:13.556236 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.556196 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01ec765f-ea51-44ac-b38c-5eb565d63f6f" path="/var/lib/kubelet/pods/01ec765f-ea51-44ac-b38c-5eb565d63f6f/volumes" Jul 29 20:49:13.646167 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.646133 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.646370 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.646174 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f05ba6f3-359e-4f72-82da-edae6eb4df21-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.646370 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.646197 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-config-volume\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.646370 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.646213 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.646370 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.646285 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f05ba6f3-359e-4f72-82da-edae6eb4df21-config-out\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.646370 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.646352 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f05ba6f3-359e-4f72-82da-edae6eb4df21-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.646600 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.646401 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-web-config\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.646600 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.646435 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk5zd\" (UniqueName: \"kubernetes.io/projected/f05ba6f3-359e-4f72-82da-edae6eb4df21-kube-api-access-bk5zd\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.646600 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.646503 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f05ba6f3-359e-4f72-82da-edae6eb4df21-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.646600 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.646530 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.646600 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.646556 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f05ba6f3-359e-4f72-82da-edae6eb4df21-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.646600 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.646581 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.646797 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.646652 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.747983 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.747876 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f05ba6f3-359e-4f72-82da-edae6eb4df21-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.747983 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.747914 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.748192 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.747988 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.748192 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.748024 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.748192 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.748044 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f05ba6f3-359e-4f72-82da-edae6eb4df21-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.748192 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.748062 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-config-volume\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.748192 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.748082 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.748192 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.748115 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f05ba6f3-359e-4f72-82da-edae6eb4df21-config-out\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.748192 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.748145 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f05ba6f3-359e-4f72-82da-edae6eb4df21-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.748192 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.748189 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-web-config\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.748641 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.748214 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk5zd\" (UniqueName: \"kubernetes.io/projected/f05ba6f3-359e-4f72-82da-edae6eb4df21-kube-api-access-bk5zd\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.748641 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.748258 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f05ba6f3-359e-4f72-82da-edae6eb4df21-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.748641 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.748292 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.748641 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.748364 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f05ba6f3-359e-4f72-82da-edae6eb4df21-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.748857 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.748832 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f05ba6f3-359e-4f72-82da-edae6eb4df21-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.751541 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.751243 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f05ba6f3-359e-4f72-82da-edae6eb4df21-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.751541 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.751244 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.751541 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.751386 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.751541 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.751430 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.751861 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.751766 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-config-volume\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.751861 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.751817 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.752178 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.752157 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.752314 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.752292 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f05ba6f3-359e-4f72-82da-edae6eb4df21-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.753415 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.753388 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f05ba6f3-359e-4f72-82da-edae6eb4df21-web-config\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.753582 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.753564 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f05ba6f3-359e-4f72-82da-edae6eb4df21-config-out\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.759256 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.759233 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk5zd\" (UniqueName: \"kubernetes.io/projected/f05ba6f3-359e-4f72-82da-edae6eb4df21-kube-api-access-bk5zd\") pod \"alertmanager-main-0\" (UID: \"f05ba6f3-359e-4f72-82da-edae6eb4df21\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.787107 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.787073 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 20:49:13.920524 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:13.920498 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 20:49:13.922559 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:49:13.922535 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05ba6f3_359e_4f72_82da_edae6eb4df21.slice/crio-5c902bdaf95482367058d4b37dd1093372225dbfca525813bfb8f9001044cd10 WatchSource:0}: Error finding container 5c902bdaf95482367058d4b37dd1093372225dbfca525813bfb8f9001044cd10: Status 404 returned error can't find the container with id 5c902bdaf95482367058d4b37dd1093372225dbfca525813bfb8f9001044cd10 Jul 29 20:49:14.433963 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:14.433905 2562 generic.go:356] "Generic (PLEG): container finished" podID="f05ba6f3-359e-4f72-82da-edae6eb4df21" containerID="cac9a0e500fc7ac508e6e291f1b52fa83ec10f5f764252e53e482c0509311ada" exitCode=0 Jul 29 20:49:14.434364 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:14.433993 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f05ba6f3-359e-4f72-82da-edae6eb4df21","Type":"ContainerDied","Data":"cac9a0e500fc7ac508e6e291f1b52fa83ec10f5f764252e53e482c0509311ada"} Jul 29 20:49:14.434364 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:14.434108 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f05ba6f3-359e-4f72-82da-edae6eb4df21","Type":"ContainerStarted","Data":"5c902bdaf95482367058d4b37dd1093372225dbfca525813bfb8f9001044cd10"} Jul 29 20:49:15.441651 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:15.441574 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f05ba6f3-359e-4f72-82da-edae6eb4df21","Type":"ContainerStarted","Data":"2dc738d3ccefd2da9aa2bf5480efd1edf91357f9e3a899c3cee027be56fc2e3f"} Jul 29 20:49:15.441651 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:15.441641 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f05ba6f3-359e-4f72-82da-edae6eb4df21","Type":"ContainerStarted","Data":"2f274ef00177e0c279c8353892d35af605dcf5701a466cd282c99c8c390497bc"} Jul 29 20:49:15.442057 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:15.441667 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f05ba6f3-359e-4f72-82da-edae6eb4df21","Type":"ContainerStarted","Data":"343f59c81efbf65c7a28646303c44ee45f36f5010588a85ba27d64a6ce1937f5"} Jul 29 20:49:15.442057 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:15.441695 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f05ba6f3-359e-4f72-82da-edae6eb4df21","Type":"ContainerStarted","Data":"659627fbfbad86aae172a7d7aa6c6e3eca77f686838ab298f5a3dc2c27878b97"} Jul 29 20:49:15.442057 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:15.441731 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f05ba6f3-359e-4f72-82da-edae6eb4df21","Type":"ContainerStarted","Data":"1edbdb66b9543329e30ab6a9b39f6ee4952ef1459b5a36945824ec7388826824"} Jul 29 20:49:15.442057 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:15.441758 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f05ba6f3-359e-4f72-82da-edae6eb4df21","Type":"ContainerStarted","Data":"c0265ebeb9addf27e1b76b40b4cbb6f8c2dbb23ffbff80b140e1ab51c993724c"} Jul 29 20:49:15.467663 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:15.467607 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.452461343 podStartE2EDuration="2.467591928s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.130585ms" isStatefulPod=false podCreationTimestamp="2026-07-29 20:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:49:15.466736841 +0000 UTC m=+172.337840310" watchObservedRunningTime="2026-07-29 20:49:15.467591928 +0000 UTC m=+172.338695394" Jul 29 20:49:19.565953 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:19.565906 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:19.566372 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:19.565992 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:19.570809 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:19.570786 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:20.467405 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:20.467369 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:49:20.511713 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:20.511685 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66784977cd-dw2zf"] Jul 29 20:49:45.534760 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:45.534715 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66784977cd-dw2zf" podUID="2f20f2ec-d69d-47cd-8948-0e78ba4c0995" containerName="console" containerID="cri-o://0cf9499e41ba7224f51105f88b83b0fe221dd284900d85566a0c5622412d24bd" gracePeriod=15 Jul 29 20:49:45.779153 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:45.779124 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:49:45.934932 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:45.934881 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-service-ca\") pod \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " Jul 29 20:49:45.935118 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:45.934966 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-console-config\") pod \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " Jul 29 20:49:45.935118 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:45.935050 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l9bp\" (UniqueName: \"kubernetes.io/projected/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-kube-api-access-2l9bp\") pod \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " Jul 29 20:49:45.935118 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:45.935096 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-trusted-ca-bundle\") pod \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " Jul 29 20:49:45.935228 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:45.935149 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-oauth-serving-cert\") pod \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " Jul 29 20:49:45.935228 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:45.935172 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-console-serving-cert\") pod \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " Jul 29 20:49:45.935228 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:45.935199 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-console-oauth-config\") pod \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\" (UID: \"2f20f2ec-d69d-47cd-8948-0e78ba4c0995\") " Jul 29 20:49:45.935381 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:45.935266 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-service-ca" (OuterVolumeSpecName: "service-ca") pod "2f20f2ec-d69d-47cd-8948-0e78ba4c0995" (UID: "2f20f2ec-d69d-47cd-8948-0e78ba4c0995"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:49:45.935476 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:45.935450 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-console-config" (OuterVolumeSpecName: "console-config") pod "2f20f2ec-d69d-47cd-8948-0e78ba4c0995" (UID: "2f20f2ec-d69d-47cd-8948-0e78ba4c0995"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:49:45.935543 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:45.935515 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-service-ca\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:45.935607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:45.935562 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2f20f2ec-d69d-47cd-8948-0e78ba4c0995" (UID: "2f20f2ec-d69d-47cd-8948-0e78ba4c0995"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:49:45.935668 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:45.935593 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2f20f2ec-d69d-47cd-8948-0e78ba4c0995" (UID: "2f20f2ec-d69d-47cd-8948-0e78ba4c0995"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:49:45.937203 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:45.937177 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-kube-api-access-2l9bp" (OuterVolumeSpecName: "kube-api-access-2l9bp") pod "2f20f2ec-d69d-47cd-8948-0e78ba4c0995" (UID: "2f20f2ec-d69d-47cd-8948-0e78ba4c0995"). InnerVolumeSpecName "kube-api-access-2l9bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:49:45.937345 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:45.937322 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2f20f2ec-d69d-47cd-8948-0e78ba4c0995" (UID: "2f20f2ec-d69d-47cd-8948-0e78ba4c0995"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:49:45.937391 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:45.937355 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2f20f2ec-d69d-47cd-8948-0e78ba4c0995" (UID: "2f20f2ec-d69d-47cd-8948-0e78ba4c0995"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:49:46.035982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:46.035911 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l9bp\" (UniqueName: \"kubernetes.io/projected/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-kube-api-access-2l9bp\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:46.035982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:46.035973 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-trusted-ca-bundle\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:46.035982 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:46.035983 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-oauth-serving-cert\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:46.036208 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:46.035998 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-console-serving-cert\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:46.036208 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:46.036007 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-console-oauth-config\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:46.036208 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:46.036015 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f20f2ec-d69d-47cd-8948-0e78ba4c0995-console-config\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:49:46.576465 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:46.576428 2562 generic.go:356] "Generic (PLEG): container finished" podID="2f20f2ec-d69d-47cd-8948-0e78ba4c0995" containerID="0cf9499e41ba7224f51105f88b83b0fe221dd284900d85566a0c5622412d24bd" exitCode=0 Jul 29 20:49:46.576861 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:46.576468 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66784977cd-dw2zf" Jul 29 20:49:46.576861 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:46.576510 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66784977cd-dw2zf" event={"ID":"2f20f2ec-d69d-47cd-8948-0e78ba4c0995","Type":"ContainerDied","Data":"0cf9499e41ba7224f51105f88b83b0fe221dd284900d85566a0c5622412d24bd"} Jul 29 20:49:46.576861 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:46.576561 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66784977cd-dw2zf" event={"ID":"2f20f2ec-d69d-47cd-8948-0e78ba4c0995","Type":"ContainerDied","Data":"677e4827d4ea856d15e4e4839f4c4f557d6eeb39c11023d668881b9ecfbff48f"} Jul 29 20:49:46.576861 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:46.576581 2562 scope.go:117] "RemoveContainer" containerID="0cf9499e41ba7224f51105f88b83b0fe221dd284900d85566a0c5622412d24bd" Jul 29 20:49:46.584805 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:46.584779 2562 scope.go:117] "RemoveContainer" containerID="0cf9499e41ba7224f51105f88b83b0fe221dd284900d85566a0c5622412d24bd" Jul 29 20:49:46.585064 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:49:46.585045 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf9499e41ba7224f51105f88b83b0fe221dd284900d85566a0c5622412d24bd\": container with ID starting with 0cf9499e41ba7224f51105f88b83b0fe221dd284900d85566a0c5622412d24bd not found: ID does not exist" containerID="0cf9499e41ba7224f51105f88b83b0fe221dd284900d85566a0c5622412d24bd" Jul 29 20:49:46.585137 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:46.585072 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf9499e41ba7224f51105f88b83b0fe221dd284900d85566a0c5622412d24bd"} err="failed to get container status \"0cf9499e41ba7224f51105f88b83b0fe221dd284900d85566a0c5622412d24bd\": rpc error: code = NotFound desc = could not find container \"0cf9499e41ba7224f51105f88b83b0fe221dd284900d85566a0c5622412d24bd\": container with ID starting with 0cf9499e41ba7224f51105f88b83b0fe221dd284900d85566a0c5622412d24bd not found: ID does not exist" Jul 29 20:49:46.595746 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:46.595720 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66784977cd-dw2zf"] Jul 29 20:49:46.599132 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:46.599112 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66784977cd-dw2zf"] Jul 29 20:49:47.555813 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:49:47.555779 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f20f2ec-d69d-47cd-8948-0e78ba4c0995" path="/var/lib/kubelet/pods/2f20f2ec-d69d-47cd-8948-0e78ba4c0995/volumes" Jul 29 20:50:09.295403 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.295371 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5"] Jul 29 20:50:09.295830 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.295685 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f20f2ec-d69d-47cd-8948-0e78ba4c0995" containerName="console" Jul 29 20:50:09.295830 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.295695 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f20f2ec-d69d-47cd-8948-0e78ba4c0995" containerName="console" Jul 29 20:50:09.295830 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.295745 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f20f2ec-d69d-47cd-8948-0e78ba4c0995" containerName="console" Jul 29 20:50:09.300079 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.300062 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5" Jul 29 20:50:09.302223 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.302203 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 20:50:09.302329 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.302203 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 20:50:09.302655 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.302637 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xpc47\"" Jul 29 20:50:09.307631 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.307584 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5"] Jul 29 20:50:09.338933 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.338880 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6af42b1-85c4-485b-80cd-38eb8d584ec1-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5\" (UID: \"c6af42b1-85c4-485b-80cd-38eb8d584ec1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5" Jul 29 20:50:09.339096 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.338952 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6af42b1-85c4-485b-80cd-38eb8d584ec1-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5\" (UID: \"c6af42b1-85c4-485b-80cd-38eb8d584ec1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5" Jul 29 20:50:09.339096 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.338971 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxqn8\" (UniqueName: \"kubernetes.io/projected/c6af42b1-85c4-485b-80cd-38eb8d584ec1-kube-api-access-xxqn8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5\" (UID: \"c6af42b1-85c4-485b-80cd-38eb8d584ec1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5" Jul 29 20:50:09.439971 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.439904 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6af42b1-85c4-485b-80cd-38eb8d584ec1-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5\" (UID: \"c6af42b1-85c4-485b-80cd-38eb8d584ec1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5" Jul 29 20:50:09.439971 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.439975 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxqn8\" (UniqueName: \"kubernetes.io/projected/c6af42b1-85c4-485b-80cd-38eb8d584ec1-kube-api-access-xxqn8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5\" (UID: \"c6af42b1-85c4-485b-80cd-38eb8d584ec1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5" Jul 29 20:50:09.440199 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.440048 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6af42b1-85c4-485b-80cd-38eb8d584ec1-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5\" (UID: \"c6af42b1-85c4-485b-80cd-38eb8d584ec1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5" Jul 29 20:50:09.440333 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.440311 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6af42b1-85c4-485b-80cd-38eb8d584ec1-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5\" (UID: \"c6af42b1-85c4-485b-80cd-38eb8d584ec1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5" Jul 29 20:50:09.440372 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.440334 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6af42b1-85c4-485b-80cd-38eb8d584ec1-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5\" (UID: \"c6af42b1-85c4-485b-80cd-38eb8d584ec1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5" Jul 29 20:50:09.447488 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.447460 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxqn8\" (UniqueName: \"kubernetes.io/projected/c6af42b1-85c4-485b-80cd-38eb8d584ec1-kube-api-access-xxqn8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5\" (UID: \"c6af42b1-85c4-485b-80cd-38eb8d584ec1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5" Jul 29 20:50:09.610709 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.610656 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5" Jul 29 20:50:09.743115 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:09.743087 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5"] Jul 29 20:50:09.745375 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:50:09.745345 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6af42b1_85c4_485b_80cd_38eb8d584ec1.slice/crio-05e9971441ceda6119b74e07ec24fe4c944e0a20eb201aefb57ee6cfbc3d8fdf WatchSource:0}: Error finding container 05e9971441ceda6119b74e07ec24fe4c944e0a20eb201aefb57ee6cfbc3d8fdf: Status 404 returned error can't find the container with id 05e9971441ceda6119b74e07ec24fe4c944e0a20eb201aefb57ee6cfbc3d8fdf Jul 29 20:50:10.675723 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:10.675669 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5" event={"ID":"c6af42b1-85c4-485b-80cd-38eb8d584ec1","Type":"ContainerStarted","Data":"05e9971441ceda6119b74e07ec24fe4c944e0a20eb201aefb57ee6cfbc3d8fdf"} Jul 29 20:50:17.717794 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:17.717756 2562 generic.go:356] "Generic (PLEG): container finished" podID="c6af42b1-85c4-485b-80cd-38eb8d584ec1" containerID="c73e64febe6f779560f18006a1dedb41c3321beda214b160076d7bd265afea42" exitCode=0 Jul 29 20:50:17.718258 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:17.717840 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5" event={"ID":"c6af42b1-85c4-485b-80cd-38eb8d584ec1","Type":"ContainerDied","Data":"c73e64febe6f779560f18006a1dedb41c3321beda214b160076d7bd265afea42"} Jul 29 20:50:19.729569 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:19.729527 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5" event={"ID":"c6af42b1-85c4-485b-80cd-38eb8d584ec1","Type":"ContainerStarted","Data":"40eb0103fd60d9db223fff9d579c9c5c722684baa530ba992026a4733fc99861"} Jul 29 20:50:20.735265 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:20.735229 2562 generic.go:356] "Generic (PLEG): container finished" podID="c6af42b1-85c4-485b-80cd-38eb8d584ec1" containerID="40eb0103fd60d9db223fff9d579c9c5c722684baa530ba992026a4733fc99861" exitCode=0 Jul 29 20:50:20.735708 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:20.735318 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5" event={"ID":"c6af42b1-85c4-485b-80cd-38eb8d584ec1","Type":"ContainerDied","Data":"40eb0103fd60d9db223fff9d579c9c5c722684baa530ba992026a4733fc99861"} Jul 29 20:50:27.772545 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:27.772505 2562 generic.go:356] "Generic (PLEG): container finished" podID="c6af42b1-85c4-485b-80cd-38eb8d584ec1" containerID="3358c92f175ff72964cb89af622cad82c67c876a3836004b4d4234444f25db83" exitCode=0 Jul 29 20:50:27.772949 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:27.772584 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5" event={"ID":"c6af42b1-85c4-485b-80cd-38eb8d584ec1","Type":"ContainerDied","Data":"3358c92f175ff72964cb89af622cad82c67c876a3836004b4d4234444f25db83"} Jul 29 20:50:28.904040 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:28.904013 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5" Jul 29 20:50:29.008998 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:29.008958 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxqn8\" (UniqueName: \"kubernetes.io/projected/c6af42b1-85c4-485b-80cd-38eb8d584ec1-kube-api-access-xxqn8\") pod \"c6af42b1-85c4-485b-80cd-38eb8d584ec1\" (UID: \"c6af42b1-85c4-485b-80cd-38eb8d584ec1\") " Jul 29 20:50:29.008998 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:29.009005 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6af42b1-85c4-485b-80cd-38eb8d584ec1-util\") pod \"c6af42b1-85c4-485b-80cd-38eb8d584ec1\" (UID: \"c6af42b1-85c4-485b-80cd-38eb8d584ec1\") " Jul 29 20:50:29.009228 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:29.009027 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6af42b1-85c4-485b-80cd-38eb8d584ec1-bundle\") pod \"c6af42b1-85c4-485b-80cd-38eb8d584ec1\" (UID: \"c6af42b1-85c4-485b-80cd-38eb8d584ec1\") " Jul 29 20:50:29.009734 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:29.009709 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6af42b1-85c4-485b-80cd-38eb8d584ec1-bundle" (OuterVolumeSpecName: "bundle") pod "c6af42b1-85c4-485b-80cd-38eb8d584ec1" (UID: "c6af42b1-85c4-485b-80cd-38eb8d584ec1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:50:29.011255 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:29.011232 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6af42b1-85c4-485b-80cd-38eb8d584ec1-kube-api-access-xxqn8" (OuterVolumeSpecName: "kube-api-access-xxqn8") pod "c6af42b1-85c4-485b-80cd-38eb8d584ec1" (UID: "c6af42b1-85c4-485b-80cd-38eb8d584ec1"). InnerVolumeSpecName "kube-api-access-xxqn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:50:29.013368 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:29.013343 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6af42b1-85c4-485b-80cd-38eb8d584ec1-util" (OuterVolumeSpecName: "util") pod "c6af42b1-85c4-485b-80cd-38eb8d584ec1" (UID: "c6af42b1-85c4-485b-80cd-38eb8d584ec1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:50:29.110263 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:29.110229 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxqn8\" (UniqueName: \"kubernetes.io/projected/c6af42b1-85c4-485b-80cd-38eb8d584ec1-kube-api-access-xxqn8\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:50:29.110263 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:29.110262 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6af42b1-85c4-485b-80cd-38eb8d584ec1-util\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:50:29.110454 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:29.110272 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6af42b1-85c4-485b-80cd-38eb8d584ec1-bundle\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:50:29.782421 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:29.782386 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5" event={"ID":"c6af42b1-85c4-485b-80cd-38eb8d584ec1","Type":"ContainerDied","Data":"05e9971441ceda6119b74e07ec24fe4c944e0a20eb201aefb57ee6cfbc3d8fdf"} Jul 29 20:50:29.782638 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:29.782438 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05e9971441ceda6119b74e07ec24fe4c944e0a20eb201aefb57ee6cfbc3d8fdf" Jul 29 20:50:29.782638 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:29.782395 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjwz5" Jul 29 20:50:37.105929 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.105877 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-68whm"] Jul 29 20:50:37.106387 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.106238 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6af42b1-85c4-485b-80cd-38eb8d584ec1" containerName="pull" Jul 29 20:50:37.106387 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.106250 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6af42b1-85c4-485b-80cd-38eb8d584ec1" containerName="pull" Jul 29 20:50:37.106387 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.106261 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6af42b1-85c4-485b-80cd-38eb8d584ec1" containerName="util" Jul 29 20:50:37.106387 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.106270 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6af42b1-85c4-485b-80cd-38eb8d584ec1" containerName="util" Jul 29 20:50:37.106387 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.106285 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6af42b1-85c4-485b-80cd-38eb8d584ec1" containerName="extract" Jul 29 20:50:37.106387 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.106291 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6af42b1-85c4-485b-80cd-38eb8d584ec1" containerName="extract" Jul 29 20:50:37.106387 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.106350 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6af42b1-85c4-485b-80cd-38eb8d584ec1" containerName="extract" Jul 29 20:50:37.110066 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.110045 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-68whm" Jul 29 20:50:37.111879 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.111858 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 29 20:50:37.112027 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.111905 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 29 20:50:37.112027 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.111934 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-rj9wd\"" Jul 29 20:50:37.120713 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.120689 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-68whm"] Jul 29 20:50:37.178671 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.178633 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl9kr\" (UniqueName: \"kubernetes.io/projected/12ff5cdc-5aee-4326-a4e5-56f086390306-kube-api-access-sl9kr\") pod \"cert-manager-operator-controller-manager-55c444bf86-68whm\" (UID: \"12ff5cdc-5aee-4326-a4e5-56f086390306\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-68whm" Jul 29 20:50:37.178671 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.178674 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12ff5cdc-5aee-4326-a4e5-56f086390306-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-68whm\" (UID: \"12ff5cdc-5aee-4326-a4e5-56f086390306\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-68whm" Jul 29 20:50:37.279661 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.279624 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl9kr\" (UniqueName: \"kubernetes.io/projected/12ff5cdc-5aee-4326-a4e5-56f086390306-kube-api-access-sl9kr\") pod \"cert-manager-operator-controller-manager-55c444bf86-68whm\" (UID: \"12ff5cdc-5aee-4326-a4e5-56f086390306\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-68whm" Jul 29 20:50:37.279661 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.279664 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12ff5cdc-5aee-4326-a4e5-56f086390306-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-68whm\" (UID: \"12ff5cdc-5aee-4326-a4e5-56f086390306\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-68whm" Jul 29 20:50:37.280072 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.280049 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12ff5cdc-5aee-4326-a4e5-56f086390306-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-68whm\" (UID: \"12ff5cdc-5aee-4326-a4e5-56f086390306\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-68whm" Jul 29 20:50:37.289624 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.289593 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl9kr\" (UniqueName: \"kubernetes.io/projected/12ff5cdc-5aee-4326-a4e5-56f086390306-kube-api-access-sl9kr\") pod \"cert-manager-operator-controller-manager-55c444bf86-68whm\" (UID: \"12ff5cdc-5aee-4326-a4e5-56f086390306\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-68whm" Jul 29 20:50:37.419613 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.419527 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-68whm" Jul 29 20:50:37.553905 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:50:37.553869 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ff5cdc_5aee_4326_a4e5_56f086390306.slice/crio-62bafee13b99473f37acc4de5c7532304bd25684bca1ec0a3c9bf3b8786e107f WatchSource:0}: Error finding container 62bafee13b99473f37acc4de5c7532304bd25684bca1ec0a3c9bf3b8786e107f: Status 404 returned error can't find the container with id 62bafee13b99473f37acc4de5c7532304bd25684bca1ec0a3c9bf3b8786e107f Jul 29 20:50:37.556681 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.556553 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-68whm"] Jul 29 20:50:37.818779 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:37.818742 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-68whm" event={"ID":"12ff5cdc-5aee-4326-a4e5-56f086390306","Type":"ContainerStarted","Data":"62bafee13b99473f37acc4de5c7532304bd25684bca1ec0a3c9bf3b8786e107f"} Jul 29 20:50:39.830194 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:39.830103 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-68whm" event={"ID":"12ff5cdc-5aee-4326-a4e5-56f086390306","Type":"ContainerStarted","Data":"8ffc02299ecfcf2c9aa57b579a8a052b2e0a69669df98c5947a0123d9139ff2e"} Jul 29 20:50:39.848409 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:39.848357 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-68whm" podStartSLOduration=0.847944551 podStartE2EDuration="2.848340101s" totalImagesPullingTime="2.00039555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:50:39.846223742 +0000 UTC m=+256.717327228" watchObservedRunningTime="2026-07-29 20:50:39.848340101 +0000 UTC m=+256.719443568" Jul 29 20:50:41.161561 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.161524 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt"] Jul 29 20:50:41.164172 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.164149 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt" Jul 29 20:50:41.166136 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.166114 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 20:50:41.166382 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.166366 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 20:50:41.166435 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.166382 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xpc47\"" Jul 29 20:50:41.173943 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.173900 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt"] Jul 29 20:50:41.310968 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.310899 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt\" (UID: \"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt" Jul 29 20:50:41.311129 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.311014 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4szd6\" (UniqueName: \"kubernetes.io/projected/82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32-kube-api-access-4szd6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt\" (UID: \"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt" Jul 29 20:50:41.311129 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.311038 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt\" (UID: \"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt" Jul 29 20:50:41.412234 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.412149 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4szd6\" (UniqueName: \"kubernetes.io/projected/82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32-kube-api-access-4szd6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt\" (UID: \"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt" Jul 29 20:50:41.412234 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.412189 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt\" (UID: \"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt" Jul 29 20:50:41.412234 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.412228 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt\" (UID: \"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt" Jul 29 20:50:41.412548 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.412526 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt\" (UID: \"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt" Jul 29 20:50:41.412593 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.412538 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt\" (UID: \"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt" Jul 29 20:50:41.419010 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.418987 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4szd6\" (UniqueName: \"kubernetes.io/projected/82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32-kube-api-access-4szd6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt\" (UID: \"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt" Jul 29 20:50:41.474140 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.474097 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt" Jul 29 20:50:41.598983 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.598956 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt"] Jul 29 20:50:41.601469 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:50:41.601445 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e9f094_d8a1_4e3d_91b9_0b3a9bd7ab32.slice/crio-3ff5cd05b8b94bace84b977bc1072eeb7ba512818eca61776b9b9411e935ee97 WatchSource:0}: Error finding container 3ff5cd05b8b94bace84b977bc1072eeb7ba512818eca61776b9b9411e935ee97: Status 404 returned error can't find the container with id 3ff5cd05b8b94bace84b977bc1072eeb7ba512818eca61776b9b9411e935ee97 Jul 29 20:50:41.845422 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.845382 2562 generic.go:356] "Generic (PLEG): container finished" podID="82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32" containerID="452c7a4ed972435c9c8496ef5f64587f69a7ffdec4c0d1dad2d7678b7180a6c0" exitCode=0 Jul 29 20:50:41.845619 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.845478 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt" event={"ID":"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32","Type":"ContainerDied","Data":"452c7a4ed972435c9c8496ef5f64587f69a7ffdec4c0d1dad2d7678b7180a6c0"} Jul 29 20:50:41.845619 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:41.845522 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt" event={"ID":"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32","Type":"ContainerStarted","Data":"3ff5cd05b8b94bace84b977bc1072eeb7ba512818eca61776b9b9411e935ee97"} Jul 29 20:50:42.926656 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:42.926621 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vsctk"] Jul 29 20:50:42.928825 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:42.928801 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsctk" Jul 29 20:50:42.930593 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:42.930571 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 29 20:50:42.930718 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:42.930610 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 29 20:50:42.931049 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:42.931033 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-jrn7t\"" Jul 29 20:50:42.936661 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:42.936564 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vsctk"] Jul 29 20:50:43.029401 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:43.029367 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsxqg\" (UniqueName: \"kubernetes.io/projected/ab46b7a5-e033-43ca-8033-c44799f85328-kube-api-access-zsxqg\") pod \"cert-manager-cainjector-559cdd8c87-vsctk\" (UID: \"ab46b7a5-e033-43ca-8033-c44799f85328\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsctk" Jul 29 20:50:43.029401 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:43.029403 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab46b7a5-e033-43ca-8033-c44799f85328-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vsctk\" (UID: \"ab46b7a5-e033-43ca-8033-c44799f85328\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsctk" Jul 29 20:50:43.129913 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:43.129877 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsxqg\" (UniqueName: \"kubernetes.io/projected/ab46b7a5-e033-43ca-8033-c44799f85328-kube-api-access-zsxqg\") pod \"cert-manager-cainjector-559cdd8c87-vsctk\" (UID: \"ab46b7a5-e033-43ca-8033-c44799f85328\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsctk" Jul 29 20:50:43.129913 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:43.129936 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab46b7a5-e033-43ca-8033-c44799f85328-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vsctk\" (UID: \"ab46b7a5-e033-43ca-8033-c44799f85328\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsctk" Jul 29 20:50:43.138046 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:43.138011 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab46b7a5-e033-43ca-8033-c44799f85328-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vsctk\" (UID: \"ab46b7a5-e033-43ca-8033-c44799f85328\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsctk" Jul 29 20:50:43.138345 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:43.138325 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsxqg\" (UniqueName: \"kubernetes.io/projected/ab46b7a5-e033-43ca-8033-c44799f85328-kube-api-access-zsxqg\") pod \"cert-manager-cainjector-559cdd8c87-vsctk\" (UID: \"ab46b7a5-e033-43ca-8033-c44799f85328\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsctk" Jul 29 20:50:43.247956 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:43.247861 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsctk" Jul 29 20:50:43.808573 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:43.808419 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vsctk"] Jul 29 20:50:43.810726 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:50:43.810698 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab46b7a5_e033_43ca_8033_c44799f85328.slice/crio-33b58790df89007a4aef4b84fb8200a334ee3cb0e930e80044953deb4feab6a6 WatchSource:0}: Error finding container 33b58790df89007a4aef4b84fb8200a334ee3cb0e930e80044953deb4feab6a6: Status 404 returned error can't find the container with id 33b58790df89007a4aef4b84fb8200a334ee3cb0e930e80044953deb4feab6a6 Jul 29 20:50:43.857799 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:43.857769 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt" event={"ID":"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32","Type":"ContainerStarted","Data":"0a8cec0a535f7c9925bbf7380d7ef55cff3faa4008ec32173ba2a65db789c4cd"} Jul 29 20:50:43.858999 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:43.858977 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsctk" event={"ID":"ab46b7a5-e033-43ca-8033-c44799f85328","Type":"ContainerStarted","Data":"33b58790df89007a4aef4b84fb8200a334ee3cb0e930e80044953deb4feab6a6"} Jul 29 20:50:44.866150 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:44.866108 2562 generic.go:356] "Generic (PLEG): container finished" podID="82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32" containerID="0a8cec0a535f7c9925bbf7380d7ef55cff3faa4008ec32173ba2a65db789c4cd" exitCode=0 Jul 29 20:50:44.866150 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:44.866147 2562 generic.go:356] "Generic (PLEG): container finished" podID="82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32" containerID="6574a1c3a41e24ed798c682674b380d4b40ad506c17871de60d2f37956014e8b" exitCode=0 Jul 29 20:50:44.866696 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:44.866201 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt" event={"ID":"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32","Type":"ContainerDied","Data":"0a8cec0a535f7c9925bbf7380d7ef55cff3faa4008ec32173ba2a65db789c4cd"} Jul 29 20:50:44.866696 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:44.866285 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt" event={"ID":"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32","Type":"ContainerDied","Data":"6574a1c3a41e24ed798c682674b380d4b40ad506c17871de60d2f37956014e8b"} Jul 29 20:50:46.752148 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:46.752123 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt" Jul 29 20:50:46.865470 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:46.865421 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4szd6\" (UniqueName: \"kubernetes.io/projected/82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32-kube-api-access-4szd6\") pod \"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32\" (UID: \"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32\") " Jul 29 20:50:46.865635 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:46.865486 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32-bundle\") pod \"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32\" (UID: \"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32\") " Jul 29 20:50:46.865635 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:46.865630 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32-util\") pod \"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32\" (UID: \"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32\") " Jul 29 20:50:46.866161 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:46.866129 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32-bundle" (OuterVolumeSpecName: "bundle") pod "82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32" (UID: "82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:50:46.867779 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:46.867751 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32-kube-api-access-4szd6" (OuterVolumeSpecName: "kube-api-access-4szd6") pod "82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32" (UID: "82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32"). InnerVolumeSpecName "kube-api-access-4szd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:50:46.878234 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:46.878210 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt" Jul 29 20:50:46.878349 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:46.878212 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frmqnt" event={"ID":"82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32","Type":"ContainerDied","Data":"3ff5cd05b8b94bace84b977bc1072eeb7ba512818eca61776b9b9411e935ee97"} Jul 29 20:50:46.878349 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:46.878341 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ff5cd05b8b94bace84b977bc1072eeb7ba512818eca61776b9b9411e935ee97" Jul 29 20:50:46.879649 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:46.879623 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsctk" event={"ID":"ab46b7a5-e033-43ca-8033-c44799f85328","Type":"ContainerStarted","Data":"a8ab3ef222ddc5582adba11115537b8e92dc8091c847e8e3cb6745fcfd08f46a"} Jul 29 20:50:46.894897 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:46.894756 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsctk" podStartSLOduration=1.913951145 podStartE2EDuration="4.894733947s" totalImagesPullingTime="2.980782802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:50:46.893963839 +0000 UTC m=+263.765067309" watchObservedRunningTime="2026-07-29 20:50:46.894733947 +0000 UTC m=+263.765837414" Jul 29 20:50:46.960611 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:46.960556 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32-util" (OuterVolumeSpecName: "util") pod "82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32" (UID: "82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:50:46.967224 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:46.967197 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32-util\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:50:46.967224 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:46.967223 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4szd6\" (UniqueName: \"kubernetes.io/projected/82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32-kube-api-access-4szd6\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:50:46.967390 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:46.967311 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32-bundle\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:50:51.507250 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:51.507215 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xtk5m"] Jul 29 20:50:51.507716 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:51.507544 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32" containerName="pull" Jul 29 20:50:51.507716 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:51.507555 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32" containerName="pull" Jul 29 20:50:51.507716 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:51.507569 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32" containerName="extract" Jul 29 20:50:51.507716 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:51.507575 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32" containerName="extract" Jul 29 20:50:51.507716 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:51.507589 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32" containerName="util" Jul 29 20:50:51.507716 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:51.507595 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32" containerName="util" Jul 29 20:50:51.507716 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:51.507646 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="82e9f094-d8a1-4e3d-91b9-0b3a9bd7ab32" containerName="extract" Jul 29 20:50:51.511523 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:51.511506 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xtk5m" Jul 29 20:50:51.513580 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:51.513560 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-5t9sc\"" Jul 29 20:50:51.519104 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:51.519081 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xtk5m"] Jul 29 20:50:51.608787 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:51.608747 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb6c9eaf-d55f-43a1-834b-8afa6b6da0d4-bound-sa-token\") pod \"cert-manager-55b4d954d5-xtk5m\" (UID: \"bb6c9eaf-d55f-43a1-834b-8afa6b6da0d4\") " pod="cert-manager/cert-manager-55b4d954d5-xtk5m" Jul 29 20:50:51.609006 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:51.608817 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gqfq\" (UniqueName: \"kubernetes.io/projected/bb6c9eaf-d55f-43a1-834b-8afa6b6da0d4-kube-api-access-6gqfq\") pod \"cert-manager-55b4d954d5-xtk5m\" (UID: \"bb6c9eaf-d55f-43a1-834b-8afa6b6da0d4\") " pod="cert-manager/cert-manager-55b4d954d5-xtk5m" Jul 29 20:50:51.709746 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:51.709711 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gqfq\" (UniqueName: \"kubernetes.io/projected/bb6c9eaf-d55f-43a1-834b-8afa6b6da0d4-kube-api-access-6gqfq\") pod \"cert-manager-55b4d954d5-xtk5m\" (UID: \"bb6c9eaf-d55f-43a1-834b-8afa6b6da0d4\") " pod="cert-manager/cert-manager-55b4d954d5-xtk5m" Jul 29 20:50:51.709891 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:51.709828 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb6c9eaf-d55f-43a1-834b-8afa6b6da0d4-bound-sa-token\") pod \"cert-manager-55b4d954d5-xtk5m\" (UID: \"bb6c9eaf-d55f-43a1-834b-8afa6b6da0d4\") " pod="cert-manager/cert-manager-55b4d954d5-xtk5m" Jul 29 20:50:51.719113 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:51.719090 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb6c9eaf-d55f-43a1-834b-8afa6b6da0d4-bound-sa-token\") pod \"cert-manager-55b4d954d5-xtk5m\" (UID: \"bb6c9eaf-d55f-43a1-834b-8afa6b6da0d4\") " pod="cert-manager/cert-manager-55b4d954d5-xtk5m" Jul 29 20:50:51.719276 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:51.719231 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gqfq\" (UniqueName: \"kubernetes.io/projected/bb6c9eaf-d55f-43a1-834b-8afa6b6da0d4-kube-api-access-6gqfq\") pod \"cert-manager-55b4d954d5-xtk5m\" (UID: \"bb6c9eaf-d55f-43a1-834b-8afa6b6da0d4\") " pod="cert-manager/cert-manager-55b4d954d5-xtk5m" Jul 29 20:50:51.821419 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:51.821387 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xtk5m" Jul 29 20:50:51.946198 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:51.946174 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xtk5m"] Jul 29 20:50:51.948208 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:50:51.948181 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb6c9eaf_d55f_43a1_834b_8afa6b6da0d4.slice/crio-04ca087d79cee1c8aa9ad8adaab2b11daa8baeccd7e38510372e5dcc45816cce WatchSource:0}: Error finding container 04ca087d79cee1c8aa9ad8adaab2b11daa8baeccd7e38510372e5dcc45816cce: Status 404 returned error can't find the container with id 04ca087d79cee1c8aa9ad8adaab2b11daa8baeccd7e38510372e5dcc45816cce Jul 29 20:50:52.908782 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:52.908746 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xtk5m" event={"ID":"bb6c9eaf-d55f-43a1-834b-8afa6b6da0d4","Type":"ContainerStarted","Data":"b2bb10696cc9106a8cb1a072ec005a58a8b60d887240754f975d412a1008d6bd"} Jul 29 20:50:52.909182 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:52.908789 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xtk5m" event={"ID":"bb6c9eaf-d55f-43a1-834b-8afa6b6da0d4","Type":"ContainerStarted","Data":"04ca087d79cee1c8aa9ad8adaab2b11daa8baeccd7e38510372e5dcc45816cce"} Jul 29 20:50:53.749690 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:53.749627 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-xtk5m" podStartSLOduration=2.749603311 podStartE2EDuration="2.749603311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:50:52.92251738 +0000 UTC m=+269.793620848" watchObservedRunningTime="2026-07-29 20:50:53.749603311 +0000 UTC m=+270.620706778" Jul 29 20:50:53.750331 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:53.750309 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8hmzs"] Jul 29 20:50:53.754686 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:53.754660 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8hmzs" Jul 29 20:50:53.756552 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:53.756531 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6dlc2\"" Jul 29 20:50:53.757102 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:53.757083 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 29 20:50:53.757376 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:53.757087 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 29 20:50:53.759182 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:53.759160 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8hmzs"] Jul 29 20:50:53.930366 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:53.930324 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b786c126-dc63-4b7c-8dd3-7ecefc1c7f33-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8hmzs\" (UID: \"b786c126-dc63-4b7c-8dd3-7ecefc1c7f33\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8hmzs" Jul 29 20:50:53.930768 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:53.930462 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qckz9\" (UniqueName: \"kubernetes.io/projected/b786c126-dc63-4b7c-8dd3-7ecefc1c7f33-kube-api-access-qckz9\") pod \"openshift-lws-operator-fd8ccff4c-8hmzs\" (UID: \"b786c126-dc63-4b7c-8dd3-7ecefc1c7f33\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8hmzs" Jul 29 20:50:54.031798 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:54.031716 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b786c126-dc63-4b7c-8dd3-7ecefc1c7f33-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8hmzs\" (UID: \"b786c126-dc63-4b7c-8dd3-7ecefc1c7f33\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8hmzs" Jul 29 20:50:54.032008 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:54.031817 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qckz9\" (UniqueName: \"kubernetes.io/projected/b786c126-dc63-4b7c-8dd3-7ecefc1c7f33-kube-api-access-qckz9\") pod \"openshift-lws-operator-fd8ccff4c-8hmzs\" (UID: \"b786c126-dc63-4b7c-8dd3-7ecefc1c7f33\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8hmzs" Jul 29 20:50:54.032164 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:54.032138 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b786c126-dc63-4b7c-8dd3-7ecefc1c7f33-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8hmzs\" (UID: \"b786c126-dc63-4b7c-8dd3-7ecefc1c7f33\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8hmzs" Jul 29 20:50:54.038668 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:54.038642 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qckz9\" (UniqueName: \"kubernetes.io/projected/b786c126-dc63-4b7c-8dd3-7ecefc1c7f33-kube-api-access-qckz9\") pod \"openshift-lws-operator-fd8ccff4c-8hmzs\" (UID: \"b786c126-dc63-4b7c-8dd3-7ecefc1c7f33\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8hmzs" Jul 29 20:50:54.064525 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:54.064483 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8hmzs" Jul 29 20:50:54.196658 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:54.196623 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8hmzs"] Jul 29 20:50:54.199430 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:50:54.199392 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb786c126_dc63_4b7c_8dd3_7ecefc1c7f33.slice/crio-bd2b17c76159ff55b4fed58e5a7088c2c25661f2d56da354b9061138c15d8a87 WatchSource:0}: Error finding container bd2b17c76159ff55b4fed58e5a7088c2c25661f2d56da354b9061138c15d8a87: Status 404 returned error can't find the container with id bd2b17c76159ff55b4fed58e5a7088c2c25661f2d56da354b9061138c15d8a87 Jul 29 20:50:54.921479 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:54.921431 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8hmzs" event={"ID":"b786c126-dc63-4b7c-8dd3-7ecefc1c7f33","Type":"ContainerStarted","Data":"bd2b17c76159ff55b4fed58e5a7088c2c25661f2d56da354b9061138c15d8a87"} Jul 29 20:50:56.935303 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:56.935263 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8hmzs" event={"ID":"b786c126-dc63-4b7c-8dd3-7ecefc1c7f33","Type":"ContainerStarted","Data":"1c8fa0e3e839e7d4d9cf498742b775b9be33569f8b4ee3d27a11db49f59ffdda"} Jul 29 20:50:56.948714 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:50:56.948661 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8hmzs" podStartSLOduration=1.7107504489999998 podStartE2EDuration="3.94864318s" totalImagesPullingTime="2.237892731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:50:56.948396693 +0000 UTC m=+273.819500160" watchObservedRunningTime="2026-07-29 20:50:56.94864318 +0000 UTC m=+273.819746648" Jul 29 20:51:01.609600 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:01.609562 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb"] Jul 29 20:51:01.612226 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:01.612207 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb" Jul 29 20:51:01.613968 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:01.613910 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xpc47\"" Jul 29 20:51:01.614073 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:01.614009 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 20:51:01.614368 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:01.614350 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 20:51:01.621356 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:01.621328 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb"] Jul 29 20:51:01.695909 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:01.695865 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0394525-0b90-41b7-95b8-df2b90420474-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb\" (UID: \"c0394525-0b90-41b7-95b8-df2b90420474\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb" Jul 29 20:51:01.696097 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:01.696031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7hmn\" (UniqueName: \"kubernetes.io/projected/c0394525-0b90-41b7-95b8-df2b90420474-kube-api-access-c7hmn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb\" (UID: \"c0394525-0b90-41b7-95b8-df2b90420474\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb" Jul 29 20:51:01.696097 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:01.696062 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0394525-0b90-41b7-95b8-df2b90420474-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb\" (UID: \"c0394525-0b90-41b7-95b8-df2b90420474\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb" Jul 29 20:51:01.796934 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:01.796892 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0394525-0b90-41b7-95b8-df2b90420474-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb\" (UID: \"c0394525-0b90-41b7-95b8-df2b90420474\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb" Jul 29 20:51:01.797192 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:01.797016 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7hmn\" (UniqueName: \"kubernetes.io/projected/c0394525-0b90-41b7-95b8-df2b90420474-kube-api-access-c7hmn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb\" (UID: \"c0394525-0b90-41b7-95b8-df2b90420474\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb" Jul 29 20:51:01.797192 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:01.797044 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0394525-0b90-41b7-95b8-df2b90420474-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb\" (UID: \"c0394525-0b90-41b7-95b8-df2b90420474\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb" Jul 29 20:51:01.797356 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:01.797333 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0394525-0b90-41b7-95b8-df2b90420474-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb\" (UID: \"c0394525-0b90-41b7-95b8-df2b90420474\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb" Jul 29 20:51:01.797395 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:01.797361 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0394525-0b90-41b7-95b8-df2b90420474-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb\" (UID: \"c0394525-0b90-41b7-95b8-df2b90420474\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb" Jul 29 20:51:01.805421 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:01.805388 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7hmn\" (UniqueName: \"kubernetes.io/projected/c0394525-0b90-41b7-95b8-df2b90420474-kube-api-access-c7hmn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb\" (UID: \"c0394525-0b90-41b7-95b8-df2b90420474\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb" Jul 29 20:51:01.922414 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:01.922331 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb" Jul 29 20:51:02.048323 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:02.048296 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb"] Jul 29 20:51:02.050013 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:51:02.049980 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0394525_0b90_41b7_95b8_df2b90420474.slice/crio-0787aeb091101c1ae7e9c9d1e74ad06134031e9950d8a3792b4e23a1f10bfef5 WatchSource:0}: Error finding container 0787aeb091101c1ae7e9c9d1e74ad06134031e9950d8a3792b4e23a1f10bfef5: Status 404 returned error can't find the container with id 0787aeb091101c1ae7e9c9d1e74ad06134031e9950d8a3792b4e23a1f10bfef5 Jul 29 20:51:02.963835 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:02.963800 2562 generic.go:356] "Generic (PLEG): container finished" podID="c0394525-0b90-41b7-95b8-df2b90420474" containerID="7392399eeb39872dc3870864c694b4fe8749cb0906e153d621dcb49e95ad8e65" exitCode=0 Jul 29 20:51:02.964269 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:02.963900 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb" event={"ID":"c0394525-0b90-41b7-95b8-df2b90420474","Type":"ContainerDied","Data":"7392399eeb39872dc3870864c694b4fe8749cb0906e153d621dcb49e95ad8e65"} Jul 29 20:51:02.964269 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:02.963975 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb" event={"ID":"c0394525-0b90-41b7-95b8-df2b90420474","Type":"ContainerStarted","Data":"0787aeb091101c1ae7e9c9d1e74ad06134031e9950d8a3792b4e23a1f10bfef5"} Jul 29 20:51:03.970649 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:03.970553 2562 generic.go:356] "Generic (PLEG): container finished" podID="c0394525-0b90-41b7-95b8-df2b90420474" containerID="cb3c285de4673c63e4a93628f33301985945f8a0d783745dde25135902c32886" exitCode=0 Jul 29 20:51:03.970649 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:03.970602 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb" event={"ID":"c0394525-0b90-41b7-95b8-df2b90420474","Type":"ContainerDied","Data":"cb3c285de4673c63e4a93628f33301985945f8a0d783745dde25135902c32886"} Jul 29 20:51:04.977560 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:04.977519 2562 generic.go:356] "Generic (PLEG): container finished" podID="c0394525-0b90-41b7-95b8-df2b90420474" containerID="a9a3bd42470136bd293ab82169193dd10d32d552863ff7bb7f4235d6a922eb22" exitCode=0 Jul 29 20:51:04.977967 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:04.977567 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb" event={"ID":"c0394525-0b90-41b7-95b8-df2b90420474","Type":"ContainerDied","Data":"a9a3bd42470136bd293ab82169193dd10d32d552863ff7bb7f4235d6a922eb22"} Jul 29 20:51:06.113552 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:06.113522 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb" Jul 29 20:51:06.128852 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:06.128491 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0394525-0b90-41b7-95b8-df2b90420474-bundle\") pod \"c0394525-0b90-41b7-95b8-df2b90420474\" (UID: \"c0394525-0b90-41b7-95b8-df2b90420474\") " Jul 29 20:51:06.128852 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:06.128547 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7hmn\" (UniqueName: \"kubernetes.io/projected/c0394525-0b90-41b7-95b8-df2b90420474-kube-api-access-c7hmn\") pod \"c0394525-0b90-41b7-95b8-df2b90420474\" (UID: \"c0394525-0b90-41b7-95b8-df2b90420474\") " Jul 29 20:51:06.128852 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:06.128664 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0394525-0b90-41b7-95b8-df2b90420474-util\") pod \"c0394525-0b90-41b7-95b8-df2b90420474\" (UID: \"c0394525-0b90-41b7-95b8-df2b90420474\") " Jul 29 20:51:06.129885 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:06.129843 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0394525-0b90-41b7-95b8-df2b90420474-bundle" (OuterVolumeSpecName: "bundle") pod "c0394525-0b90-41b7-95b8-df2b90420474" (UID: "c0394525-0b90-41b7-95b8-df2b90420474"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:51:06.137526 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:06.137474 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0394525-0b90-41b7-95b8-df2b90420474-util" (OuterVolumeSpecName: "util") pod "c0394525-0b90-41b7-95b8-df2b90420474" (UID: "c0394525-0b90-41b7-95b8-df2b90420474"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:51:06.137659 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:06.137617 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0394525-0b90-41b7-95b8-df2b90420474-kube-api-access-c7hmn" (OuterVolumeSpecName: "kube-api-access-c7hmn") pod "c0394525-0b90-41b7-95b8-df2b90420474" (UID: "c0394525-0b90-41b7-95b8-df2b90420474"). InnerVolumeSpecName "kube-api-access-c7hmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:51:06.229452 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:06.229416 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0394525-0b90-41b7-95b8-df2b90420474-bundle\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:51:06.229452 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:06.229446 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7hmn\" (UniqueName: \"kubernetes.io/projected/c0394525-0b90-41b7-95b8-df2b90420474-kube-api-access-c7hmn\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:51:06.229452 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:06.229456 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0394525-0b90-41b7-95b8-df2b90420474-util\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:51:06.989572 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:06.989535 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb" event={"ID":"c0394525-0b90-41b7-95b8-df2b90420474","Type":"ContainerDied","Data":"0787aeb091101c1ae7e9c9d1e74ad06134031e9950d8a3792b4e23a1f10bfef5"} Jul 29 20:51:06.989753 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:06.989591 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0787aeb091101c1ae7e9c9d1e74ad06134031e9950d8a3792b4e23a1f10bfef5" Jul 29 20:51:06.989753 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:06.989547 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc7rqb" Jul 29 20:51:14.724205 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.724152 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm"] Jul 29 20:51:14.724607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.724483 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0394525-0b90-41b7-95b8-df2b90420474" containerName="extract" Jul 29 20:51:14.724607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.724495 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0394525-0b90-41b7-95b8-df2b90420474" containerName="extract" Jul 29 20:51:14.724607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.724506 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0394525-0b90-41b7-95b8-df2b90420474" containerName="util" Jul 29 20:51:14.724607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.724513 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0394525-0b90-41b7-95b8-df2b90420474" containerName="util" Jul 29 20:51:14.724607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.724530 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0394525-0b90-41b7-95b8-df2b90420474" containerName="pull" Jul 29 20:51:14.724607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.724536 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0394525-0b90-41b7-95b8-df2b90420474" containerName="pull" Jul 29 20:51:14.724607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.724590 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0394525-0b90-41b7-95b8-df2b90420474" containerName="extract" Jul 29 20:51:14.727063 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.727041 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" Jul 29 20:51:14.729146 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.729124 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 29 20:51:14.729146 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.729135 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-x7p5x\"" Jul 29 20:51:14.729325 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.729150 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 29 20:51:14.729325 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.729221 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 29 20:51:14.729491 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.729477 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 29 20:51:14.744949 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.744905 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm"] Jul 29 20:51:14.806021 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.805984 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzzc9\" (UniqueName: \"kubernetes.io/projected/85f40ff7-a510-4ece-bec3-665fac570daa-kube-api-access-zzzc9\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-dxxkm\" (UID: \"85f40ff7-a510-4ece-bec3-665fac570daa\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" Jul 29 20:51:14.806206 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.806081 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/85f40ff7-a510-4ece-bec3-665fac570daa-manifests\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-dxxkm\" (UID: \"85f40ff7-a510-4ece-bec3-665fac570daa\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" Jul 29 20:51:14.806206 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.806132 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85f40ff7-a510-4ece-bec3-665fac570daa-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-dxxkm\" (UID: \"85f40ff7-a510-4ece-bec3-665fac570daa\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" Jul 29 20:51:14.806295 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.806238 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/85f40ff7-a510-4ece-bec3-665fac570daa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-dxxkm\" (UID: \"85f40ff7-a510-4ece-bec3-665fac570daa\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" Jul 29 20:51:14.806295 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.806280 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85f40ff7-a510-4ece-bec3-665fac570daa-tmp\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-dxxkm\" (UID: \"85f40ff7-a510-4ece-bec3-665fac570daa\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" Jul 29 20:51:14.907347 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.907298 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/85f40ff7-a510-4ece-bec3-665fac570daa-manifests\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-dxxkm\" (UID: \"85f40ff7-a510-4ece-bec3-665fac570daa\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" Jul 29 20:51:14.907347 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.907350 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85f40ff7-a510-4ece-bec3-665fac570daa-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-dxxkm\" (UID: \"85f40ff7-a510-4ece-bec3-665fac570daa\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" Jul 29 20:51:14.907720 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.907391 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/85f40ff7-a510-4ece-bec3-665fac570daa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-dxxkm\" (UID: \"85f40ff7-a510-4ece-bec3-665fac570daa\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" Jul 29 20:51:14.907720 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.907423 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85f40ff7-a510-4ece-bec3-665fac570daa-tmp\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-dxxkm\" (UID: \"85f40ff7-a510-4ece-bec3-665fac570daa\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" Jul 29 20:51:14.907720 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.907462 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzzc9\" (UniqueName: \"kubernetes.io/projected/85f40ff7-a510-4ece-bec3-665fac570daa-kube-api-access-zzzc9\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-dxxkm\" (UID: \"85f40ff7-a510-4ece-bec3-665fac570daa\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" Jul 29 20:51:14.907888 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.907849 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/85f40ff7-a510-4ece-bec3-665fac570daa-manifests\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-dxxkm\" (UID: \"85f40ff7-a510-4ece-bec3-665fac570daa\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" Jul 29 20:51:14.909877 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.909850 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85f40ff7-a510-4ece-bec3-665fac570daa-tmp\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-dxxkm\" (UID: \"85f40ff7-a510-4ece-bec3-665fac570daa\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" Jul 29 20:51:14.910098 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.910076 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85f40ff7-a510-4ece-bec3-665fac570daa-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-dxxkm\" (UID: \"85f40ff7-a510-4ece-bec3-665fac570daa\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" Jul 29 20:51:14.910152 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.910097 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/85f40ff7-a510-4ece-bec3-665fac570daa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-dxxkm\" (UID: \"85f40ff7-a510-4ece-bec3-665fac570daa\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" Jul 29 20:51:14.915500 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:14.915474 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzzc9\" (UniqueName: \"kubernetes.io/projected/85f40ff7-a510-4ece-bec3-665fac570daa-kube-api-access-zzzc9\") pod \"opendatahub-operator-controller-manager-7dbc94cc7-dxxkm\" (UID: \"85f40ff7-a510-4ece-bec3-665fac570daa\") " pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" Jul 29 20:51:15.038039 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:15.037933 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" Jul 29 20:51:15.176399 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:15.176373 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm"] Jul 29 20:51:15.178403 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:51:15.178370 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f40ff7_a510_4ece_bec3_665fac570daa.slice/crio-a007fb084254647c12ba49511636901080471f69374fe7beb54316a60fe60d2d WatchSource:0}: Error finding container a007fb084254647c12ba49511636901080471f69374fe7beb54316a60fe60d2d: Status 404 returned error can't find the container with id a007fb084254647c12ba49511636901080471f69374fe7beb54316a60fe60d2d Jul 29 20:51:16.038838 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:16.038803 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" event={"ID":"85f40ff7-a510-4ece-bec3-665fac570daa","Type":"ContainerStarted","Data":"a007fb084254647c12ba49511636901080471f69374fe7beb54316a60fe60d2d"} Jul 29 20:51:19.056188 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:19.056149 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" event={"ID":"85f40ff7-a510-4ece-bec3-665fac570daa","Type":"ContainerStarted","Data":"538667f537280db81f26f1a686627dac9fa20e2e1553c3a37798d8eb23b662c7"} Jul 29 20:51:20.062875 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:20.062834 2562 generic.go:356] "Generic (PLEG): container finished" podID="85f40ff7-a510-4ece-bec3-665fac570daa" containerID="538667f537280db81f26f1a686627dac9fa20e2e1553c3a37798d8eb23b662c7" exitCode=0 Jul 29 20:51:20.063313 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:20.062940 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" event={"ID":"85f40ff7-a510-4ece-bec3-665fac570daa","Type":"ContainerDied","Data":"538667f537280db81f26f1a686627dac9fa20e2e1553c3a37798d8eb23b662c7"} Jul 29 20:51:21.069976 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:21.069940 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" event={"ID":"85f40ff7-a510-4ece-bec3-665fac570daa","Type":"ContainerStarted","Data":"4df8f494e24f7493403eb42935438edbe2a9a2815ccc288ccd695ed3f7154707"} Jul 29 20:51:21.070395 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:21.070057 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" Jul 29 20:51:21.091580 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:21.091523 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" podStartSLOduration=2.675060776 podStartE2EDuration="7.091508239s" totalImagesPullingTime="3.310649277s" totalInitContainerRuntime="1.105798186s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:51:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:51:21.09000402 +0000 UTC m=+297.961107512" watchObservedRunningTime="2026-07-29 20:51:21.091508239 +0000 UTC m=+297.962611705" Jul 29 20:51:23.477018 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:23.476989 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 20:51:23.478403 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:23.478382 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 20:51:23.490070 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:23.490046 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 20:51:32.079194 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:32.079159 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7dbc94cc7-dxxkm" Jul 29 20:51:34.506850 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.506807 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5b585ddb49-xvld7"] Jul 29 20:51:34.510786 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.510766 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b585ddb49-xvld7" Jul 29 20:51:34.514303 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.514278 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 29 20:51:34.517047 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.516982 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 29 20:51:34.533227 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.533190 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b585ddb49-xvld7"] Jul 29 20:51:34.583172 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.583138 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e97f8c98-53dd-4308-b78b-35672457368b-tls-certs\") pod \"kube-auth-proxy-5b585ddb49-xvld7\" (UID: \"e97f8c98-53dd-4308-b78b-35672457368b\") " pod="openshift-ingress/kube-auth-proxy-5b585ddb49-xvld7" Jul 29 20:51:34.583369 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.583190 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e97f8c98-53dd-4308-b78b-35672457368b-tmp\") pod \"kube-auth-proxy-5b585ddb49-xvld7\" (UID: \"e97f8c98-53dd-4308-b78b-35672457368b\") " pod="openshift-ingress/kube-auth-proxy-5b585ddb49-xvld7" Jul 29 20:51:34.583369 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.583221 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sth96\" (UniqueName: \"kubernetes.io/projected/e97f8c98-53dd-4308-b78b-35672457368b-kube-api-access-sth96\") pod \"kube-auth-proxy-5b585ddb49-xvld7\" (UID: \"e97f8c98-53dd-4308-b78b-35672457368b\") " pod="openshift-ingress/kube-auth-proxy-5b585ddb49-xvld7" Jul 29 20:51:34.684767 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.684733 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e97f8c98-53dd-4308-b78b-35672457368b-tls-certs\") pod \"kube-auth-proxy-5b585ddb49-xvld7\" (UID: \"e97f8c98-53dd-4308-b78b-35672457368b\") " pod="openshift-ingress/kube-auth-proxy-5b585ddb49-xvld7" Jul 29 20:51:34.684958 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.684791 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e97f8c98-53dd-4308-b78b-35672457368b-tmp\") pod \"kube-auth-proxy-5b585ddb49-xvld7\" (UID: \"e97f8c98-53dd-4308-b78b-35672457368b\") " pod="openshift-ingress/kube-auth-proxy-5b585ddb49-xvld7" Jul 29 20:51:34.684958 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.684833 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sth96\" (UniqueName: \"kubernetes.io/projected/e97f8c98-53dd-4308-b78b-35672457368b-kube-api-access-sth96\") pod \"kube-auth-proxy-5b585ddb49-xvld7\" (UID: \"e97f8c98-53dd-4308-b78b-35672457368b\") " pod="openshift-ingress/kube-auth-proxy-5b585ddb49-xvld7" Jul 29 20:51:34.687026 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.686995 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e97f8c98-53dd-4308-b78b-35672457368b-tmp\") pod \"kube-auth-proxy-5b585ddb49-xvld7\" (UID: \"e97f8c98-53dd-4308-b78b-35672457368b\") " pod="openshift-ingress/kube-auth-proxy-5b585ddb49-xvld7" Jul 29 20:51:34.687280 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.687258 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e97f8c98-53dd-4308-b78b-35672457368b-tls-certs\") pod \"kube-auth-proxy-5b585ddb49-xvld7\" (UID: \"e97f8c98-53dd-4308-b78b-35672457368b\") " pod="openshift-ingress/kube-auth-proxy-5b585ddb49-xvld7" Jul 29 20:51:34.694709 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.694660 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sth96\" (UniqueName: \"kubernetes.io/projected/e97f8c98-53dd-4308-b78b-35672457368b-kube-api-access-sth96\") pod \"kube-auth-proxy-5b585ddb49-xvld7\" (UID: \"e97f8c98-53dd-4308-b78b-35672457368b\") " pod="openshift-ingress/kube-auth-proxy-5b585ddb49-xvld7" Jul 29 20:51:34.762007 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.761907 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6s7ss"] Jul 29 20:51:34.766240 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.766215 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-6s7ss" Jul 29 20:51:34.768271 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.768249 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 29 20:51:34.768403 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.768354 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-pgkjj\"" Jul 29 20:51:34.776877 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.776847 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6s7ss"] Jul 29 20:51:34.821267 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.821195 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b585ddb49-xvld7" Jul 29 20:51:34.886659 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.886630 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjjfm\" (UniqueName: \"kubernetes.io/projected/1eeae0db-d5e9-4447-a622-8415e6b733a3-kube-api-access-xjjfm\") pod \"odh-model-controller-7998c67dc4-6s7ss\" (UID: \"1eeae0db-d5e9-4447-a622-8415e6b733a3\") " pod="opendatahub/odh-model-controller-7998c67dc4-6s7ss" Jul 29 20:51:34.886766 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.886674 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1eeae0db-d5e9-4447-a622-8415e6b733a3-cert\") pod \"odh-model-controller-7998c67dc4-6s7ss\" (UID: \"1eeae0db-d5e9-4447-a622-8415e6b733a3\") " pod="opendatahub/odh-model-controller-7998c67dc4-6s7ss" Jul 29 20:51:34.969314 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.969287 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b585ddb49-xvld7"] Jul 29 20:51:34.971264 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:51:34.971238 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97f8c98_53dd_4308_b78b_35672457368b.slice/crio-bac122c8084aec05bce1df2bacb4e279c763cae3d8654d75492be5d290f6ca3c WatchSource:0}: Error finding container bac122c8084aec05bce1df2bacb4e279c763cae3d8654d75492be5d290f6ca3c: Status 404 returned error can't find the container with id bac122c8084aec05bce1df2bacb4e279c763cae3d8654d75492be5d290f6ca3c Jul 29 20:51:34.973445 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.973429 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 20:51:34.987825 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.987799 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjjfm\" (UniqueName: \"kubernetes.io/projected/1eeae0db-d5e9-4447-a622-8415e6b733a3-kube-api-access-xjjfm\") pod \"odh-model-controller-7998c67dc4-6s7ss\" (UID: \"1eeae0db-d5e9-4447-a622-8415e6b733a3\") " pod="opendatahub/odh-model-controller-7998c67dc4-6s7ss" Jul 29 20:51:34.987983 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.987840 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1eeae0db-d5e9-4447-a622-8415e6b733a3-cert\") pod \"odh-model-controller-7998c67dc4-6s7ss\" (UID: \"1eeae0db-d5e9-4447-a622-8415e6b733a3\") " pod="opendatahub/odh-model-controller-7998c67dc4-6s7ss" Jul 29 20:51:34.987983 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:51:34.987975 2562 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 29 20:51:34.988117 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:51:34.988029 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eeae0db-d5e9-4447-a622-8415e6b733a3-cert podName:1eeae0db-d5e9-4447-a622-8415e6b733a3 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:35.488012706 +0000 UTC m=+312.359116151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1eeae0db-d5e9-4447-a622-8415e6b733a3-cert") pod "odh-model-controller-7998c67dc4-6s7ss" (UID: "1eeae0db-d5e9-4447-a622-8415e6b733a3") : secret "odh-model-controller-webhook-cert" not found Jul 29 20:51:34.998904 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:34.998870 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjjfm\" (UniqueName: \"kubernetes.io/projected/1eeae0db-d5e9-4447-a622-8415e6b733a3-kube-api-access-xjjfm\") pod \"odh-model-controller-7998c67dc4-6s7ss\" (UID: \"1eeae0db-d5e9-4447-a622-8415e6b733a3\") " pod="opendatahub/odh-model-controller-7998c67dc4-6s7ss" Jul 29 20:51:35.145217 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.145173 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b585ddb49-xvld7" event={"ID":"e97f8c98-53dd-4308-b78b-35672457368b","Type":"ContainerStarted","Data":"bac122c8084aec05bce1df2bacb4e279c763cae3d8654d75492be5d290f6ca3c"} Jul 29 20:51:35.493244 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.493157 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1eeae0db-d5e9-4447-a622-8415e6b733a3-cert\") pod \"odh-model-controller-7998c67dc4-6s7ss\" (UID: \"1eeae0db-d5e9-4447-a622-8415e6b733a3\") " pod="opendatahub/odh-model-controller-7998c67dc4-6s7ss" Jul 29 20:51:35.495846 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.495821 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1eeae0db-d5e9-4447-a622-8415e6b733a3-cert\") pod \"odh-model-controller-7998c67dc4-6s7ss\" (UID: \"1eeae0db-d5e9-4447-a622-8415e6b733a3\") " pod="opendatahub/odh-model-controller-7998c67dc4-6s7ss" Jul 29 20:51:35.666688 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.666647 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bjtgx"] Jul 29 20:51:35.670310 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.670282 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bjtgx" Jul 29 20:51:35.672304 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.672274 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 29 20:51:35.672434 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.672322 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-dvsl9\"" Jul 29 20:51:35.678988 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.678957 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-6s7ss" Jul 29 20:51:35.684169 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.684143 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bjtgx"] Jul 29 20:51:35.796489 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.796425 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a870fec-1f88-41c2-8bf6-e239cbac331c-tls-certs\") pod \"model-serving-api-54d89bd5fc-bjtgx\" (UID: \"5a870fec-1f88-41c2-8bf6-e239cbac331c\") " pod="opendatahub/model-serving-api-54d89bd5fc-bjtgx" Jul 29 20:51:35.796668 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.796637 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2cdw\" (UniqueName: \"kubernetes.io/projected/5a870fec-1f88-41c2-8bf6-e239cbac331c-kube-api-access-g2cdw\") pod \"model-serving-api-54d89bd5fc-bjtgx\" (UID: \"5a870fec-1f88-41c2-8bf6-e239cbac331c\") " pod="opendatahub/model-serving-api-54d89bd5fc-bjtgx" Jul 29 20:51:35.816464 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.816421 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz"] Jul 29 20:51:35.822516 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.822059 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:35.824942 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.823808 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 29 20:51:35.824942 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.824176 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 29 20:51:35.824942 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.824407 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 29 20:51:35.824942 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.824595 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-46fh7\"" Jul 29 20:51:35.824942 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.824772 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 20:51:35.832697 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.832316 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz"] Jul 29 20:51:35.897792 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.897757 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:35.898018 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.897850 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:35.898018 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.897899 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:35.898018 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.897961 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:35.898178 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.898020 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:35.898178 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.898064 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mgf4\" (UniqueName: \"kubernetes.io/projected/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-kube-api-access-2mgf4\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:35.898178 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.898107 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:35.898178 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.898153 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2cdw\" (UniqueName: \"kubernetes.io/projected/5a870fec-1f88-41c2-8bf6-e239cbac331c-kube-api-access-g2cdw\") pod \"model-serving-api-54d89bd5fc-bjtgx\" (UID: \"5a870fec-1f88-41c2-8bf6-e239cbac331c\") " pod="opendatahub/model-serving-api-54d89bd5fc-bjtgx" Jul 29 20:51:35.898414 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.898187 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a870fec-1f88-41c2-8bf6-e239cbac331c-tls-certs\") pod \"model-serving-api-54d89bd5fc-bjtgx\" (UID: \"5a870fec-1f88-41c2-8bf6-e239cbac331c\") " pod="opendatahub/model-serving-api-54d89bd5fc-bjtgx" Jul 29 20:51:35.901024 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.900970 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a870fec-1f88-41c2-8bf6-e239cbac331c-tls-certs\") pod \"model-serving-api-54d89bd5fc-bjtgx\" (UID: \"5a870fec-1f88-41c2-8bf6-e239cbac331c\") " pod="opendatahub/model-serving-api-54d89bd5fc-bjtgx" Jul 29 20:51:35.906372 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.906345 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2cdw\" (UniqueName: \"kubernetes.io/projected/5a870fec-1f88-41c2-8bf6-e239cbac331c-kube-api-access-g2cdw\") pod \"model-serving-api-54d89bd5fc-bjtgx\" (UID: \"5a870fec-1f88-41c2-8bf6-e239cbac331c\") " pod="opendatahub/model-serving-api-54d89bd5fc-bjtgx" Jul 29 20:51:35.986201 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.986159 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bjtgx" Jul 29 20:51:36.001047 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.999367 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:36.001047 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.999439 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:36.001047 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.999488 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:36.001047 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.999514 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:36.001047 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.999554 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:36.001047 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.999600 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mgf4\" (UniqueName: \"kubernetes.io/projected/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-kube-api-access-2mgf4\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:36.001047 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:35.999643 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:36.001047 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:36.000648 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:36.005671 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:36.005488 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:36.007003 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:36.006953 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:36.008168 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:36.007747 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:36.009008 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:36.008696 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:36.010771 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:36.010726 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mgf4\" (UniqueName: \"kubernetes.io/projected/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-kube-api-access-2mgf4\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:36.011122 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:36.011080 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zk8zz\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:36.051776 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:36.051736 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6s7ss"] Jul 29 20:51:36.053717 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:51:36.053682 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eeae0db_d5e9_4447_a622_8415e6b733a3.slice/crio-12aa93ac3f6a3a20874365f56676a184b13c52ea2662bb72c5134a4eb0a7142e WatchSource:0}: Error finding container 12aa93ac3f6a3a20874365f56676a184b13c52ea2662bb72c5134a4eb0a7142e: Status 404 returned error can't find the container with id 12aa93ac3f6a3a20874365f56676a184b13c52ea2662bb72c5134a4eb0a7142e Jul 29 20:51:36.140971 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:36.140912 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:36.159799 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:36.159656 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-6s7ss" event={"ID":"1eeae0db-d5e9-4447-a622-8415e6b733a3","Type":"ContainerStarted","Data":"12aa93ac3f6a3a20874365f56676a184b13c52ea2662bb72c5134a4eb0a7142e"} Jul 29 20:51:36.166892 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:36.166861 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bjtgx"] Jul 29 20:51:36.169723 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:51:36.169686 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a870fec_1f88_41c2_8bf6_e239cbac331c.slice/crio-d2eaea7ca9a3a3675d257f085d2363c38151d6d34f2abf7c16260e61f5225936 WatchSource:0}: Error finding container d2eaea7ca9a3a3675d257f085d2363c38151d6d34f2abf7c16260e61f5225936: Status 404 returned error can't find the container with id d2eaea7ca9a3a3675d257f085d2363c38151d6d34f2abf7c16260e61f5225936 Jul 29 20:51:36.309553 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:36.309522 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz"] Jul 29 20:51:36.312435 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:51:36.312403 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcaa076d_3a8f_40ce_9abd_53ec10fe2a28.slice/crio-c63b1b5e19de1aeef759477f4f91895e9e9c492e81c9ad356f17826f67d5ae33 WatchSource:0}: Error finding container c63b1b5e19de1aeef759477f4f91895e9e9c492e81c9ad356f17826f67d5ae33: Status 404 returned error can't find the container with id c63b1b5e19de1aeef759477f4f91895e9e9c492e81c9ad356f17826f67d5ae33 Jul 29 20:51:37.169536 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:37.169496 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bjtgx" event={"ID":"5a870fec-1f88-41c2-8bf6-e239cbac331c","Type":"ContainerStarted","Data":"d2eaea7ca9a3a3675d257f085d2363c38151d6d34f2abf7c16260e61f5225936"} Jul 29 20:51:37.171669 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:37.171612 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" event={"ID":"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28","Type":"ContainerStarted","Data":"c63b1b5e19de1aeef759477f4f91895e9e9c492e81c9ad356f17826f67d5ae33"} Jul 29 20:51:40.952112 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:40.952054 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 20:51:40.952530 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:40.952140 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 20:51:41.196122 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:41.196024 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b585ddb49-xvld7" event={"ID":"e97f8c98-53dd-4308-b78b-35672457368b","Type":"ContainerStarted","Data":"e7a772e12a841ae697955a6095c53739e97250b3626b586a6c49a2b17e9f7420"} Jul 29 20:51:41.197641 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:41.197613 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-6s7ss" event={"ID":"1eeae0db-d5e9-4447-a622-8415e6b733a3","Type":"ContainerStarted","Data":"62fb039ae01b0317171127a33b2c99bb7b51b903d18555ddb4d514ef2739ffed"} Jul 29 20:51:41.197791 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:41.197667 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-6s7ss" Jul 29 20:51:41.199185 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:41.199160 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bjtgx" event={"ID":"5a870fec-1f88-41c2-8bf6-e239cbac331c","Type":"ContainerStarted","Data":"49ff4558d39d3bad567c136031a9cee2996e2d9fab3cfe89cad24c61f9809446"} Jul 29 20:51:41.199322 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:41.199283 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-bjtgx" Jul 29 20:51:41.200862 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:41.200840 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" event={"ID":"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28","Type":"ContainerStarted","Data":"dc9f6df3ef655a46c866d90c68cf35937cc49b49f83c3d3e85d11f7635a00cd0"} Jul 29 20:51:41.200993 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:41.200885 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:41.213873 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:41.213823 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5b585ddb49-xvld7" podStartSLOduration=1.306416945 podStartE2EDuration="7.21380839s" totalImagesPullingTime="5.907391445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:51:41.211531861 +0000 UTC m=+318.082635329" watchObservedRunningTime="2026-07-29 20:51:41.21380839 +0000 UTC m=+318.084911856" Jul 29 20:51:41.229758 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:41.229317 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-bjtgx" podStartSLOduration=1.5249602420000001 podStartE2EDuration="6.229296027s" totalImagesPullingTime="4.704335785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:51:41.22598681 +0000 UTC m=+318.097090284" watchObservedRunningTime="2026-07-29 20:51:41.229296027 +0000 UTC m=+318.100399494" Jul 29 20:51:41.244958 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:41.244893 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" podStartSLOduration=1.6083962619999999 podStartE2EDuration="6.24487657s" totalImagesPullingTime="4.636480308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:51:41.242463831 +0000 UTC m=+318.113567298" watchObservedRunningTime="2026-07-29 20:51:41.24487657 +0000 UTC m=+318.115980036" Jul 29 20:51:41.260855 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:41.260787 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-6s7ss" podStartSLOduration=2.424202539 podStartE2EDuration="7.260765316s" totalImagesPullingTime="4.836562777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:51:41.257412598 +0000 UTC m=+318.128516069" watchObservedRunningTime="2026-07-29 20:51:41.260765316 +0000 UTC m=+318.131868785" Jul 29 20:51:42.208368 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:42.208337 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:43.888092 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:43.888052 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f"] Jul 29 20:51:43.891972 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:43.891939 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:43.902968 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:43.902940 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f"] Jul 29 20:51:43.988549 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:43.988503 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6bffcc3c-447e-46fa-bdce-d5c5af921ead-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:43.988549 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:43.988547 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6bffcc3c-447e-46fa-bdce-d5c5af921ead-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:43.988791 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:43.988640 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bffcc3c-447e-46fa-bdce-d5c5af921ead-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:43.988791 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:43.988698 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bffcc3c-447e-46fa-bdce-d5c5af921ead-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:43.988791 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:43.988742 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qvvr\" (UniqueName: \"kubernetes.io/projected/6bffcc3c-447e-46fa-bdce-d5c5af921ead-kube-api-access-2qvvr\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:43.988791 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:43.988766 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6bffcc3c-447e-46fa-bdce-d5c5af921ead-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:43.988791 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:43.988789 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6bffcc3c-447e-46fa-bdce-d5c5af921ead-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:44.090056 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:44.090008 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bffcc3c-447e-46fa-bdce-d5c5af921ead-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:44.090056 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:44.090056 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bffcc3c-447e-46fa-bdce-d5c5af921ead-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:44.090286 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:44.090090 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qvvr\" (UniqueName: \"kubernetes.io/projected/6bffcc3c-447e-46fa-bdce-d5c5af921ead-kube-api-access-2qvvr\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:44.090286 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:44.090120 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6bffcc3c-447e-46fa-bdce-d5c5af921ead-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:44.090286 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:44.090151 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6bffcc3c-447e-46fa-bdce-d5c5af921ead-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:44.090286 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:44.090209 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6bffcc3c-447e-46fa-bdce-d5c5af921ead-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:44.090286 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:44.090234 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6bffcc3c-447e-46fa-bdce-d5c5af921ead-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:44.090953 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:44.090906 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6bffcc3c-447e-46fa-bdce-d5c5af921ead-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:44.092767 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:44.092596 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6bffcc3c-447e-46fa-bdce-d5c5af921ead-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:44.092767 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:44.092667 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6bffcc3c-447e-46fa-bdce-d5c5af921ead-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:44.092767 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:44.092724 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bffcc3c-447e-46fa-bdce-d5c5af921ead-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:44.093000 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:44.092887 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6bffcc3c-447e-46fa-bdce-d5c5af921ead-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:44.097782 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:44.097759 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qvvr\" (UniqueName: \"kubernetes.io/projected/6bffcc3c-447e-46fa-bdce-d5c5af921ead-kube-api-access-2qvvr\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:44.098255 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:44.098235 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bffcc3c-447e-46fa-bdce-d5c5af921ead-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-bqn6f\" (UID: \"6bffcc3c-447e-46fa-bdce-d5c5af921ead\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:44.201664 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:44.201571 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:44.351847 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:44.351456 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f"] Jul 29 20:51:44.354103 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:51:44.354069 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bffcc3c_447e_46fa_bdce_d5c5af921ead.slice/crio-cbc449e9646500dda5ef76285ee9af2545f2570d09266339ff6c0ec175173a18 WatchSource:0}: Error finding container cbc449e9646500dda5ef76285ee9af2545f2570d09266339ff6c0ec175173a18: Status 404 returned error can't find the container with id cbc449e9646500dda5ef76285ee9af2545f2570d09266339ff6c0ec175173a18 Jul 29 20:51:44.357890 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:44.357855 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 20:51:44.358023 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:44.357937 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 20:51:45.227376 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:45.227321 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" event={"ID":"6bffcc3c-447e-46fa-bdce-d5c5af921ead","Type":"ContainerStarted","Data":"5c5b674d8029e754f6289a3f934450e1e4ba290dad340e666f930f107f8b1dc9"} Jul 29 20:51:45.227860 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:45.227397 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:45.227860 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:45.227419 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" event={"ID":"6bffcc3c-447e-46fa-bdce-d5c5af921ead","Type":"ContainerStarted","Data":"cbc449e9646500dda5ef76285ee9af2545f2570d09266339ff6c0ec175173a18"} Jul 29 20:51:45.229490 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:45.229459 2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-bqn6f container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 20:51:45.229604 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:45.229517 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" podUID="6bffcc3c-447e-46fa-bdce-d5c5af921ead" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 20:51:45.265763 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:45.265706 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" podStartSLOduration=2.265689494 podStartE2EDuration="2.265689494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:51:45.26342472 +0000 UTC m=+322.134528200" watchObservedRunningTime="2026-07-29 20:51:45.265689494 +0000 UTC m=+322.136792961" Jul 29 20:51:46.233659 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.233621 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bqn6f" Jul 29 20:51:46.287837 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.287802 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz"] Jul 29 20:51:46.288081 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.288044 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" podUID="dcaa076d-3a8f-40ce-9abd-53ec10fe2a28" containerName="discovery" containerID="cri-o://dc9f6df3ef655a46c866d90c68cf35937cc49b49f83c3d3e85d11f7635a00cd0" gracePeriod=30 Jul 29 20:51:46.539293 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.539266 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:46.718631 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.718589 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-local-certs\") pod \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " Jul 29 20:51:46.718631 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.718638 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-token\") pod \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " Jul 29 20:51:46.718895 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.718671 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-kubeconfig\") pod \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " Jul 29 20:51:46.718895 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.718703 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mgf4\" (UniqueName: \"kubernetes.io/projected/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-kube-api-access-2mgf4\") pod \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " Jul 29 20:51:46.718895 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.718727 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-csr-dns-cert\") pod \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " Jul 29 20:51:46.718895 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.718768 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-csr-ca-configmap\") pod \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " Jul 29 20:51:46.718895 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.718847 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-cacerts\") pod \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\" (UID: \"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28\") " Jul 29 20:51:46.719296 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.719255 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "dcaa076d-3a8f-40ce-9abd-53ec10fe2a28" (UID: "dcaa076d-3a8f-40ce-9abd-53ec10fe2a28"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:51:46.721270 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.721231 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "dcaa076d-3a8f-40ce-9abd-53ec10fe2a28" (UID: "dcaa076d-3a8f-40ce-9abd-53ec10fe2a28"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:51:46.721380 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.721272 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-kube-api-access-2mgf4" (OuterVolumeSpecName: "kube-api-access-2mgf4") pod "dcaa076d-3a8f-40ce-9abd-53ec10fe2a28" (UID: "dcaa076d-3a8f-40ce-9abd-53ec10fe2a28"). InnerVolumeSpecName "kube-api-access-2mgf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:51:46.721380 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.721291 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-local-certs" (OuterVolumeSpecName: "local-certs") pod "dcaa076d-3a8f-40ce-9abd-53ec10fe2a28" (UID: "dcaa076d-3a8f-40ce-9abd-53ec10fe2a28"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:51:46.721380 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.721327 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-token" (OuterVolumeSpecName: "istio-token") pod "dcaa076d-3a8f-40ce-9abd-53ec10fe2a28" (UID: "dcaa076d-3a8f-40ce-9abd-53ec10fe2a28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:51:46.721687 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.721667 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "dcaa076d-3a8f-40ce-9abd-53ec10fe2a28" (UID: "dcaa076d-3a8f-40ce-9abd-53ec10fe2a28"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:51:46.721756 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.721682 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-cacerts" (OuterVolumeSpecName: "cacerts") pod "dcaa076d-3a8f-40ce-9abd-53ec10fe2a28" (UID: "dcaa076d-3a8f-40ce-9abd-53ec10fe2a28"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:51:46.820045 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.820011 2562 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-cacerts\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:51:46.820045 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.820044 2562 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-local-certs\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:51:46.820241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.820054 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-token\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:51:46.820241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.820064 2562 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-kubeconfig\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:51:46.820241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.820072 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mgf4\" (UniqueName: \"kubernetes.io/projected/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-kube-api-access-2mgf4\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:51:46.820241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.820082 2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-csr-dns-cert\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:51:46.820241 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:46.820090 2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28-istio-csr-ca-configmap\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:51:47.240121 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:47.240026 2562 generic.go:356] "Generic (PLEG): container finished" podID="dcaa076d-3a8f-40ce-9abd-53ec10fe2a28" containerID="dc9f6df3ef655a46c866d90c68cf35937cc49b49f83c3d3e85d11f7635a00cd0" exitCode=0 Jul 29 20:51:47.240121 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:47.240071 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" Jul 29 20:51:47.240121 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:47.240106 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" event={"ID":"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28","Type":"ContainerDied","Data":"dc9f6df3ef655a46c866d90c68cf35937cc49b49f83c3d3e85d11f7635a00cd0"} Jul 29 20:51:47.240687 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:47.240162 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz" event={"ID":"dcaa076d-3a8f-40ce-9abd-53ec10fe2a28","Type":"ContainerDied","Data":"c63b1b5e19de1aeef759477f4f91895e9e9c492e81c9ad356f17826f67d5ae33"} Jul 29 20:51:47.240687 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:47.240184 2562 scope.go:117] "RemoveContainer" containerID="dc9f6df3ef655a46c866d90c68cf35937cc49b49f83c3d3e85d11f7635a00cd0" Jul 29 20:51:47.250291 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:47.250270 2562 scope.go:117] "RemoveContainer" containerID="dc9f6df3ef655a46c866d90c68cf35937cc49b49f83c3d3e85d11f7635a00cd0" Jul 29 20:51:47.250590 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:51:47.250568 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc9f6df3ef655a46c866d90c68cf35937cc49b49f83c3d3e85d11f7635a00cd0\": container with ID starting with dc9f6df3ef655a46c866d90c68cf35937cc49b49f83c3d3e85d11f7635a00cd0 not found: ID does not exist" containerID="dc9f6df3ef655a46c866d90c68cf35937cc49b49f83c3d3e85d11f7635a00cd0" Jul 29 20:51:47.250665 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:47.250600 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9f6df3ef655a46c866d90c68cf35937cc49b49f83c3d3e85d11f7635a00cd0"} err="failed to get container status \"dc9f6df3ef655a46c866d90c68cf35937cc49b49f83c3d3e85d11f7635a00cd0\": rpc error: code = NotFound desc = could not find container \"dc9f6df3ef655a46c866d90c68cf35937cc49b49f83c3d3e85d11f7635a00cd0\": container with ID starting with dc9f6df3ef655a46c866d90c68cf35937cc49b49f83c3d3e85d11f7635a00cd0 not found: ID does not exist" Jul 29 20:51:47.266135 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:47.266103 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz"] Jul 29 20:51:47.273383 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:47.273353 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zk8zz"] Jul 29 20:51:47.557476 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:47.557443 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcaa076d-3a8f-40ce-9abd-53ec10fe2a28" path="/var/lib/kubelet/pods/dcaa076d-3a8f-40ce-9abd-53ec10fe2a28/volumes" Jul 29 20:51:52.209467 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:52.209435 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-6s7ss" Jul 29 20:51:52.211491 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:51:52.211469 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-bjtgx" Jul 29 20:52:35.362332 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:35.362297 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-f4d2g"] Jul 29 20:52:35.362800 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:35.362682 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcaa076d-3a8f-40ce-9abd-53ec10fe2a28" containerName="discovery" Jul 29 20:52:35.362800 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:35.362694 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcaa076d-3a8f-40ce-9abd-53ec10fe2a28" containerName="discovery" Jul 29 20:52:35.362800 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:35.362775 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcaa076d-3a8f-40ce-9abd-53ec10fe2a28" containerName="discovery" Jul 29 20:52:35.367194 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:35.367160 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-f4d2g" Jul 29 20:52:35.369292 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:35.369265 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 20:52:35.369570 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:35.369549 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 20:52:35.369873 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:35.369855 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-9pzpc\"" Jul 29 20:52:35.375543 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:35.375507 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-f4d2g"] Jul 29 20:52:35.457322 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:35.457285 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjdhv\" (UniqueName: \"kubernetes.io/projected/c9c5cb78-be53-4c2c-bd1b-49963d9cda6e-kube-api-access-bjdhv\") pod \"kuadrant-operator-catalog-f4d2g\" (UID: \"c9c5cb78-be53-4c2c-bd1b-49963d9cda6e\") " pod="kuadrant-system/kuadrant-operator-catalog-f4d2g" Jul 29 20:52:35.558486 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:35.558454 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjdhv\" (UniqueName: \"kubernetes.io/projected/c9c5cb78-be53-4c2c-bd1b-49963d9cda6e-kube-api-access-bjdhv\") pod \"kuadrant-operator-catalog-f4d2g\" (UID: \"c9c5cb78-be53-4c2c-bd1b-49963d9cda6e\") " pod="kuadrant-system/kuadrant-operator-catalog-f4d2g" Jul 29 20:52:35.565794 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:35.565763 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjdhv\" (UniqueName: \"kubernetes.io/projected/c9c5cb78-be53-4c2c-bd1b-49963d9cda6e-kube-api-access-bjdhv\") pod \"kuadrant-operator-catalog-f4d2g\" (UID: \"c9c5cb78-be53-4c2c-bd1b-49963d9cda6e\") " pod="kuadrant-system/kuadrant-operator-catalog-f4d2g" Jul 29 20:52:35.680005 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:35.679885 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-f4d2g" Jul 29 20:52:35.727785 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:35.727748 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-f4d2g"] Jul 29 20:52:35.813336 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:35.813306 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-f4d2g"] Jul 29 20:52:35.815846 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:52:35.815809 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c5cb78_be53_4c2c_bd1b_49963d9cda6e.slice/crio-ca3acd7c5fa79bf03f060695bd005b1ae4c9237cba02f6ccc2920f3caee9da99 WatchSource:0}: Error finding container ca3acd7c5fa79bf03f060695bd005b1ae4c9237cba02f6ccc2920f3caee9da99: Status 404 returned error can't find the container with id ca3acd7c5fa79bf03f060695bd005b1ae4c9237cba02f6ccc2920f3caee9da99 Jul 29 20:52:35.937567 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:35.937489 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rg4zg"] Jul 29 20:52:35.942079 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:35.942061 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rg4zg" Jul 29 20:52:35.946904 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:35.946875 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rg4zg"] Jul 29 20:52:35.961855 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:35.961815 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjfbz\" (UniqueName: \"kubernetes.io/projected/df3a395b-ae6b-4641-93f0-8dd49fff2a81-kube-api-access-qjfbz\") pod \"kuadrant-operator-catalog-rg4zg\" (UID: \"df3a395b-ae6b-4641-93f0-8dd49fff2a81\") " pod="kuadrant-system/kuadrant-operator-catalog-rg4zg" Jul 29 20:52:36.063111 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:36.063073 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjfbz\" (UniqueName: \"kubernetes.io/projected/df3a395b-ae6b-4641-93f0-8dd49fff2a81-kube-api-access-qjfbz\") pod \"kuadrant-operator-catalog-rg4zg\" (UID: \"df3a395b-ae6b-4641-93f0-8dd49fff2a81\") " pod="kuadrant-system/kuadrant-operator-catalog-rg4zg" Jul 29 20:52:36.070534 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:36.070501 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjfbz\" (UniqueName: \"kubernetes.io/projected/df3a395b-ae6b-4641-93f0-8dd49fff2a81-kube-api-access-qjfbz\") pod \"kuadrant-operator-catalog-rg4zg\" (UID: \"df3a395b-ae6b-4641-93f0-8dd49fff2a81\") " pod="kuadrant-system/kuadrant-operator-catalog-rg4zg" Jul 29 20:52:36.253706 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:36.253618 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rg4zg" Jul 29 20:52:36.397709 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:36.397683 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rg4zg"] Jul 29 20:52:36.400893 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:52:36.400858 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3a395b_ae6b_4641_93f0_8dd49fff2a81.slice/crio-848ff759dd041b05dee52c5b780b534019fab0575d7707e7afc5343d9cb3ad23 WatchSource:0}: Error finding container 848ff759dd041b05dee52c5b780b534019fab0575d7707e7afc5343d9cb3ad23: Status 404 returned error can't find the container with id 848ff759dd041b05dee52c5b780b534019fab0575d7707e7afc5343d9cb3ad23 Jul 29 20:52:36.514652 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:36.514554 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-f4d2g" event={"ID":"c9c5cb78-be53-4c2c-bd1b-49963d9cda6e","Type":"ContainerStarted","Data":"ca3acd7c5fa79bf03f060695bd005b1ae4c9237cba02f6ccc2920f3caee9da99"} Jul 29 20:52:36.515947 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:36.515889 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rg4zg" event={"ID":"df3a395b-ae6b-4641-93f0-8dd49fff2a81","Type":"ContainerStarted","Data":"848ff759dd041b05dee52c5b780b534019fab0575d7707e7afc5343d9cb3ad23"} Jul 29 20:52:38.534442 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:38.534401 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-f4d2g" event={"ID":"c9c5cb78-be53-4c2c-bd1b-49963d9cda6e","Type":"ContainerStarted","Data":"c0b160282092c705034b099dd4341b02d3c97790f247ce9b474275e4e43ab07d"} Jul 29 20:52:38.534944 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:38.534511 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-f4d2g" podUID="c9c5cb78-be53-4c2c-bd1b-49963d9cda6e" containerName="registry-server" containerID="cri-o://c0b160282092c705034b099dd4341b02d3c97790f247ce9b474275e4e43ab07d" gracePeriod=2 Jul 29 20:52:38.536007 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:38.535968 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rg4zg" event={"ID":"df3a395b-ae6b-4641-93f0-8dd49fff2a81","Type":"ContainerStarted","Data":"c1275f0ec5f5a0bf525b5c647ae0ef26700f793389af8ed8a2cfa0f81ace96fb"} Jul 29 20:52:38.547910 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:38.547853 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-f4d2g" podStartSLOduration=1.362244194 podStartE2EDuration="3.547832267s" totalImagesPullingTime="2.185588073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:52:38.546842678 +0000 UTC m=+375.417946145" watchObservedRunningTime="2026-07-29 20:52:38.547832267 +0000 UTC m=+375.418935736" Jul 29 20:52:38.560184 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:38.560135 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-rg4zg" podStartSLOduration=1.956969169 podStartE2EDuration="3.56011658s" totalImagesPullingTime="1.603147411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:52:38.558991691 +0000 UTC m=+375.430095183" watchObservedRunningTime="2026-07-29 20:52:38.56011658 +0000 UTC m=+375.431220047" Jul 29 20:52:38.780614 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:38.780587 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-f4d2g" Jul 29 20:52:38.791685 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:38.791603 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjdhv\" (UniqueName: \"kubernetes.io/projected/c9c5cb78-be53-4c2c-bd1b-49963d9cda6e-kube-api-access-bjdhv\") pod \"c9c5cb78-be53-4c2c-bd1b-49963d9cda6e\" (UID: \"c9c5cb78-be53-4c2c-bd1b-49963d9cda6e\") " Jul 29 20:52:38.794652 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:38.794615 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c5cb78-be53-4c2c-bd1b-49963d9cda6e-kube-api-access-bjdhv" (OuterVolumeSpecName: "kube-api-access-bjdhv") pod "c9c5cb78-be53-4c2c-bd1b-49963d9cda6e" (UID: "c9c5cb78-be53-4c2c-bd1b-49963d9cda6e"). InnerVolumeSpecName "kube-api-access-bjdhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:52:38.892362 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:38.892323 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjdhv\" (UniqueName: \"kubernetes.io/projected/c9c5cb78-be53-4c2c-bd1b-49963d9cda6e-kube-api-access-bjdhv\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:52:39.542214 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:39.542181 2562 generic.go:356] "Generic (PLEG): container finished" podID="c9c5cb78-be53-4c2c-bd1b-49963d9cda6e" containerID="c0b160282092c705034b099dd4341b02d3c97790f247ce9b474275e4e43ab07d" exitCode=0 Jul 29 20:52:39.542637 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:39.542229 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-f4d2g" Jul 29 20:52:39.542637 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:39.542276 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-f4d2g" event={"ID":"c9c5cb78-be53-4c2c-bd1b-49963d9cda6e","Type":"ContainerDied","Data":"c0b160282092c705034b099dd4341b02d3c97790f247ce9b474275e4e43ab07d"} Jul 29 20:52:39.542637 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:39.542324 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-f4d2g" event={"ID":"c9c5cb78-be53-4c2c-bd1b-49963d9cda6e","Type":"ContainerDied","Data":"ca3acd7c5fa79bf03f060695bd005b1ae4c9237cba02f6ccc2920f3caee9da99"} Jul 29 20:52:39.542637 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:39.542349 2562 scope.go:117] "RemoveContainer" containerID="c0b160282092c705034b099dd4341b02d3c97790f247ce9b474275e4e43ab07d" Jul 29 20:52:39.552771 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:39.552749 2562 scope.go:117] "RemoveContainer" containerID="c0b160282092c705034b099dd4341b02d3c97790f247ce9b474275e4e43ab07d" Jul 29 20:52:39.553059 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:52:39.553038 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b160282092c705034b099dd4341b02d3c97790f247ce9b474275e4e43ab07d\": container with ID starting with c0b160282092c705034b099dd4341b02d3c97790f247ce9b474275e4e43ab07d not found: ID does not exist" containerID="c0b160282092c705034b099dd4341b02d3c97790f247ce9b474275e4e43ab07d" Jul 29 20:52:39.553141 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:39.553070 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b160282092c705034b099dd4341b02d3c97790f247ce9b474275e4e43ab07d"} err="failed to get container status \"c0b160282092c705034b099dd4341b02d3c97790f247ce9b474275e4e43ab07d\": rpc error: code = NotFound desc = could not find container \"c0b160282092c705034b099dd4341b02d3c97790f247ce9b474275e4e43ab07d\": container with ID starting with c0b160282092c705034b099dd4341b02d3c97790f247ce9b474275e4e43ab07d not found: ID does not exist" Jul 29 20:52:39.562628 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:39.562597 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-f4d2g"] Jul 29 20:52:39.570423 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:39.569448 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-f4d2g"] Jul 29 20:52:41.557036 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:41.557005 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9c5cb78-be53-4c2c-bd1b-49963d9cda6e" path="/var/lib/kubelet/pods/c9c5cb78-be53-4c2c-bd1b-49963d9cda6e/volumes" Jul 29 20:52:46.254422 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:46.254387 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-rg4zg" Jul 29 20:52:46.254845 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:46.254476 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-rg4zg" Jul 29 20:52:46.276197 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:46.276168 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-rg4zg" Jul 29 20:52:46.601162 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:46.601129 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-rg4zg" Jul 29 20:52:51.372419 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.372375 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp"] Jul 29 20:52:51.372955 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.372934 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9c5cb78-be53-4c2c-bd1b-49963d9cda6e" containerName="registry-server" Jul 29 20:52:51.373045 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.372958 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9c5cb78-be53-4c2c-bd1b-49963d9cda6e" containerName="registry-server" Jul 29 20:52:51.373103 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.373084 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9c5cb78-be53-4c2c-bd1b-49963d9cda6e" containerName="registry-server" Jul 29 20:52:51.376558 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.376524 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp" Jul 29 20:52:51.378441 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.378415 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wdbm6\"" Jul 29 20:52:51.379960 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.379896 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp"] Jul 29 20:52:51.499601 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.499562 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp\" (UID: \"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp" Jul 29 20:52:51.499792 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.499609 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lpfl\" (UniqueName: \"kubernetes.io/projected/2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c-kube-api-access-9lpfl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp\" (UID: \"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp" Jul 29 20:52:51.499792 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.499682 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp\" (UID: \"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp" Jul 29 20:52:51.600308 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.600273 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp\" (UID: \"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp" Jul 29 20:52:51.600308 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.600312 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lpfl\" (UniqueName: \"kubernetes.io/projected/2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c-kube-api-access-9lpfl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp\" (UID: \"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp" Jul 29 20:52:51.600530 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.600346 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp\" (UID: \"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp" Jul 29 20:52:51.600771 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.600747 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp\" (UID: \"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp" Jul 29 20:52:51.600822 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.600754 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp\" (UID: \"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp" Jul 29 20:52:51.609558 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.609520 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lpfl\" (UniqueName: \"kubernetes.io/projected/2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c-kube-api-access-9lpfl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp\" (UID: \"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp" Jul 29 20:52:51.686645 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.686556 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp" Jul 29 20:52:51.823793 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.823754 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp"] Jul 29 20:52:51.826914 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:52:51.826888 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd388a8_9f43_42a2_b1c1_0c45a0e77f9c.slice/crio-ee7828eafe54455b6c94a92ccdf1a16da1583dafc3663eb96ca063bb1d0df248 WatchSource:0}: Error finding container ee7828eafe54455b6c94a92ccdf1a16da1583dafc3663eb96ca063bb1d0df248: Status 404 returned error can't find the container with id ee7828eafe54455b6c94a92ccdf1a16da1583dafc3663eb96ca063bb1d0df248 Jul 29 20:52:51.970798 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.970720 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9"] Jul 29 20:52:51.974366 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.974344 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9" Jul 29 20:52:51.981523 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:51.981495 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9"] Jul 29 20:52:52.005501 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.005469 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a5db510-214d-43a6-bfdd-a6d3ad45ca01-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9\" (UID: \"9a5db510-214d-43a6-bfdd-a6d3ad45ca01\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9" Jul 29 20:52:52.005501 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.005505 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fp5b\" (UniqueName: \"kubernetes.io/projected/9a5db510-214d-43a6-bfdd-a6d3ad45ca01-kube-api-access-4fp5b\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9\" (UID: \"9a5db510-214d-43a6-bfdd-a6d3ad45ca01\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9" Jul 29 20:52:52.005690 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.005627 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a5db510-214d-43a6-bfdd-a6d3ad45ca01-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9\" (UID: \"9a5db510-214d-43a6-bfdd-a6d3ad45ca01\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9" Jul 29 20:52:52.106743 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.106703 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a5db510-214d-43a6-bfdd-a6d3ad45ca01-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9\" (UID: \"9a5db510-214d-43a6-bfdd-a6d3ad45ca01\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9" Jul 29 20:52:52.106950 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.106756 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a5db510-214d-43a6-bfdd-a6d3ad45ca01-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9\" (UID: \"9a5db510-214d-43a6-bfdd-a6d3ad45ca01\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9" Jul 29 20:52:52.106950 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.106862 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fp5b\" (UniqueName: \"kubernetes.io/projected/9a5db510-214d-43a6-bfdd-a6d3ad45ca01-kube-api-access-4fp5b\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9\" (UID: \"9a5db510-214d-43a6-bfdd-a6d3ad45ca01\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9" Jul 29 20:52:52.107180 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.107161 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a5db510-214d-43a6-bfdd-a6d3ad45ca01-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9\" (UID: \"9a5db510-214d-43a6-bfdd-a6d3ad45ca01\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9" Jul 29 20:52:52.107236 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.107166 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a5db510-214d-43a6-bfdd-a6d3ad45ca01-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9\" (UID: \"9a5db510-214d-43a6-bfdd-a6d3ad45ca01\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9" Jul 29 20:52:52.114802 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.114782 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fp5b\" (UniqueName: \"kubernetes.io/projected/9a5db510-214d-43a6-bfdd-a6d3ad45ca01-kube-api-access-4fp5b\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9\" (UID: \"9a5db510-214d-43a6-bfdd-a6d3ad45ca01\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9" Jul 29 20:52:52.292059 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.291973 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9" Jul 29 20:52:52.367968 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.367912 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt"] Jul 29 20:52:52.372813 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.372788 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt" Jul 29 20:52:52.379888 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.379821 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt"] Jul 29 20:52:52.408417 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.408380 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdfqx\" (UniqueName: \"kubernetes.io/projected/84af0a1a-df6a-405a-9b11-0fcc1f090256-kube-api-access-sdfqx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt\" (UID: \"84af0a1a-df6a-405a-9b11-0fcc1f090256\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt" Jul 29 20:52:52.408607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.408439 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84af0a1a-df6a-405a-9b11-0fcc1f090256-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt\" (UID: \"84af0a1a-df6a-405a-9b11-0fcc1f090256\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt" Jul 29 20:52:52.408607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.408508 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84af0a1a-df6a-405a-9b11-0fcc1f090256-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt\" (UID: \"84af0a1a-df6a-405a-9b11-0fcc1f090256\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt" Jul 29 20:52:52.435435 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.435408 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9"] Jul 29 20:52:52.436775 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:52:52.436746 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5db510_214d_43a6_bfdd_a6d3ad45ca01.slice/crio-9ff710471db1b43298580ca295c500c741101fe5a345d2b64e86e59b694b9351 WatchSource:0}: Error finding container 9ff710471db1b43298580ca295c500c741101fe5a345d2b64e86e59b694b9351: Status 404 returned error can't find the container with id 9ff710471db1b43298580ca295c500c741101fe5a345d2b64e86e59b694b9351 Jul 29 20:52:52.509331 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.509305 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84af0a1a-df6a-405a-9b11-0fcc1f090256-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt\" (UID: \"84af0a1a-df6a-405a-9b11-0fcc1f090256\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt" Jul 29 20:52:52.509439 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.509345 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdfqx\" (UniqueName: \"kubernetes.io/projected/84af0a1a-df6a-405a-9b11-0fcc1f090256-kube-api-access-sdfqx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt\" (UID: \"84af0a1a-df6a-405a-9b11-0fcc1f090256\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt" Jul 29 20:52:52.509439 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.509378 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84af0a1a-df6a-405a-9b11-0fcc1f090256-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt\" (UID: \"84af0a1a-df6a-405a-9b11-0fcc1f090256\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt" Jul 29 20:52:52.509675 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.509657 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84af0a1a-df6a-405a-9b11-0fcc1f090256-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt\" (UID: \"84af0a1a-df6a-405a-9b11-0fcc1f090256\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt" Jul 29 20:52:52.509716 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.509662 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84af0a1a-df6a-405a-9b11-0fcc1f090256-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt\" (UID: \"84af0a1a-df6a-405a-9b11-0fcc1f090256\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt" Jul 29 20:52:52.517169 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.517145 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdfqx\" (UniqueName: \"kubernetes.io/projected/84af0a1a-df6a-405a-9b11-0fcc1f090256-kube-api-access-sdfqx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt\" (UID: \"84af0a1a-df6a-405a-9b11-0fcc1f090256\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt" Jul 29 20:52:52.611417 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.611382 2562 generic.go:356] "Generic (PLEG): container finished" podID="2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c" containerID="3a5698f9d302cd5adf5e5abe6d6d0b2a716cb0e325c481e459408e168f5379c7" exitCode=0 Jul 29 20:52:52.611605 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.611460 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp" event={"ID":"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c","Type":"ContainerDied","Data":"3a5698f9d302cd5adf5e5abe6d6d0b2a716cb0e325c481e459408e168f5379c7"} Jul 29 20:52:52.611605 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.611509 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp" event={"ID":"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c","Type":"ContainerStarted","Data":"ee7828eafe54455b6c94a92ccdf1a16da1583dafc3663eb96ca063bb1d0df248"} Jul 29 20:52:52.613303 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.613273 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a5db510-214d-43a6-bfdd-a6d3ad45ca01" containerID="725c0585d61c3ad9ef9e6da1d4dda7b4e62228b42469368a6090a73af489396a" exitCode=0 Jul 29 20:52:52.613435 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.613316 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9" event={"ID":"9a5db510-214d-43a6-bfdd-a6d3ad45ca01","Type":"ContainerDied","Data":"725c0585d61c3ad9ef9e6da1d4dda7b4e62228b42469368a6090a73af489396a"} Jul 29 20:52:52.613435 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.613351 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9" event={"ID":"9a5db510-214d-43a6-bfdd-a6d3ad45ca01","Type":"ContainerStarted","Data":"9ff710471db1b43298580ca295c500c741101fe5a345d2b64e86e59b694b9351"} Jul 29 20:52:52.684698 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.684657 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt" Jul 29 20:52:52.776835 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.776801 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4"] Jul 29 20:52:52.781494 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.781469 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4" Jul 29 20:52:52.788133 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.788104 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4"] Jul 29 20:52:52.811939 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.811889 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9eb5e4c-6906-441e-bfdf-a1f42e282365-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4\" (UID: \"c9eb5e4c-6906-441e-bfdf-a1f42e282365\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4" Jul 29 20:52:52.812087 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.811969 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r65vq\" (UniqueName: \"kubernetes.io/projected/c9eb5e4c-6906-441e-bfdf-a1f42e282365-kube-api-access-r65vq\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4\" (UID: \"c9eb5e4c-6906-441e-bfdf-a1f42e282365\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4" Jul 29 20:52:52.812087 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.812020 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9eb5e4c-6906-441e-bfdf-a1f42e282365-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4\" (UID: \"c9eb5e4c-6906-441e-bfdf-a1f42e282365\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4" Jul 29 20:52:52.824903 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.824866 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt"] Jul 29 20:52:52.834424 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:52:52.834397 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84af0a1a_df6a_405a_9b11_0fcc1f090256.slice/crio-ffcb2cd4f4ab2583142773023b68659cd47b435a9ab99078a0274c082e3d4782 WatchSource:0}: Error finding container ffcb2cd4f4ab2583142773023b68659cd47b435a9ab99078a0274c082e3d4782: Status 404 returned error can't find the container with id ffcb2cd4f4ab2583142773023b68659cd47b435a9ab99078a0274c082e3d4782 Jul 29 20:52:52.912948 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.912895 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r65vq\" (UniqueName: \"kubernetes.io/projected/c9eb5e4c-6906-441e-bfdf-a1f42e282365-kube-api-access-r65vq\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4\" (UID: \"c9eb5e4c-6906-441e-bfdf-a1f42e282365\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4" Jul 29 20:52:52.913143 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.912992 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9eb5e4c-6906-441e-bfdf-a1f42e282365-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4\" (UID: \"c9eb5e4c-6906-441e-bfdf-a1f42e282365\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4" Jul 29 20:52:52.913143 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.913057 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9eb5e4c-6906-441e-bfdf-a1f42e282365-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4\" (UID: \"c9eb5e4c-6906-441e-bfdf-a1f42e282365\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4" Jul 29 20:52:52.913376 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.913352 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9eb5e4c-6906-441e-bfdf-a1f42e282365-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4\" (UID: \"c9eb5e4c-6906-441e-bfdf-a1f42e282365\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4" Jul 29 20:52:52.913453 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.913400 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9eb5e4c-6906-441e-bfdf-a1f42e282365-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4\" (UID: \"c9eb5e4c-6906-441e-bfdf-a1f42e282365\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4" Jul 29 20:52:52.921270 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:52.921238 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r65vq\" (UniqueName: \"kubernetes.io/projected/c9eb5e4c-6906-441e-bfdf-a1f42e282365-kube-api-access-r65vq\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4\" (UID: \"c9eb5e4c-6906-441e-bfdf-a1f42e282365\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4" Jul 29 20:52:53.095060 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:53.094969 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4" Jul 29 20:52:53.230621 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:53.230591 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4"] Jul 29 20:52:53.236634 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:52:53.236605 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9eb5e4c_6906_441e_bfdf_a1f42e282365.slice/crio-d0e3eee3cf550ddcd13829729f53d38dce1629037bbc38b227fddc4b6ed6da1e WatchSource:0}: Error finding container d0e3eee3cf550ddcd13829729f53d38dce1629037bbc38b227fddc4b6ed6da1e: Status 404 returned error can't find the container with id d0e3eee3cf550ddcd13829729f53d38dce1629037bbc38b227fddc4b6ed6da1e Jul 29 20:52:53.619509 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:53.619407 2562 generic.go:356] "Generic (PLEG): container finished" podID="84af0a1a-df6a-405a-9b11-0fcc1f090256" containerID="40396e1e2ab1f795aebc4736b94f3469bc2a9996f76f931007c19be7ababbf51" exitCode=0 Jul 29 20:52:53.620009 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:53.619508 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt" event={"ID":"84af0a1a-df6a-405a-9b11-0fcc1f090256","Type":"ContainerDied","Data":"40396e1e2ab1f795aebc4736b94f3469bc2a9996f76f931007c19be7ababbf51"} Jul 29 20:52:53.620009 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:53.619566 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt" event={"ID":"84af0a1a-df6a-405a-9b11-0fcc1f090256","Type":"ContainerStarted","Data":"ffcb2cd4f4ab2583142773023b68659cd47b435a9ab99078a0274c082e3d4782"} Jul 29 20:52:53.621743 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:53.621715 2562 generic.go:356] "Generic (PLEG): container finished" podID="2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c" containerID="a5216fb728b4c97f687d47a30b7a1c03cb731b9d7a2aeff1e3621afbd0a9b92d" exitCode=0 Jul 29 20:52:53.621957 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:53.621836 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp" event={"ID":"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c","Type":"ContainerDied","Data":"a5216fb728b4c97f687d47a30b7a1c03cb731b9d7a2aeff1e3621afbd0a9b92d"} Jul 29 20:52:53.623828 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:53.623806 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a5db510-214d-43a6-bfdd-a6d3ad45ca01" containerID="3f42f103ba21a9b106486e5ed933c867a34be5bb4d0187534af50fef1338c8fc" exitCode=0 Jul 29 20:52:53.623999 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:53.623901 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9" event={"ID":"9a5db510-214d-43a6-bfdd-a6d3ad45ca01","Type":"ContainerDied","Data":"3f42f103ba21a9b106486e5ed933c867a34be5bb4d0187534af50fef1338c8fc"} Jul 29 20:52:53.625596 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:53.625574 2562 generic.go:356] "Generic (PLEG): container finished" podID="c9eb5e4c-6906-441e-bfdf-a1f42e282365" containerID="41d1fa310ab59d9c954704875a95a97d3c0bd5d3e7b2dd31ce526839c12c0f8e" exitCode=0 Jul 29 20:52:53.625695 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:53.625599 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4" event={"ID":"c9eb5e4c-6906-441e-bfdf-a1f42e282365","Type":"ContainerDied","Data":"41d1fa310ab59d9c954704875a95a97d3c0bd5d3e7b2dd31ce526839c12c0f8e"} Jul 29 20:52:53.625695 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:53.625632 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4" event={"ID":"c9eb5e4c-6906-441e-bfdf-a1f42e282365","Type":"ContainerStarted","Data":"d0e3eee3cf550ddcd13829729f53d38dce1629037bbc38b227fddc4b6ed6da1e"} Jul 29 20:52:54.634492 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:54.634458 2562 generic.go:356] "Generic (PLEG): container finished" podID="c9eb5e4c-6906-441e-bfdf-a1f42e282365" containerID="ea2232a9b325b76e60e4bcf6c7c5f3b9bbd4224a0aa98e48ea53421d1e5216cc" exitCode=0 Jul 29 20:52:54.634996 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:54.634543 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4" event={"ID":"c9eb5e4c-6906-441e-bfdf-a1f42e282365","Type":"ContainerDied","Data":"ea2232a9b325b76e60e4bcf6c7c5f3b9bbd4224a0aa98e48ea53421d1e5216cc"} Jul 29 20:52:54.636682 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:54.636590 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt" event={"ID":"84af0a1a-df6a-405a-9b11-0fcc1f090256","Type":"ContainerStarted","Data":"78591b48a2a94a60a3f9caf2c8de270c1b1f08a9856a238ec00f74654ba2c9a9"} Jul 29 20:52:54.638848 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:54.638825 2562 generic.go:356] "Generic (PLEG): container finished" podID="2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c" containerID="ae498a635e57ad2faf8c5dbc2f7ff78b87437eb0943081391daaa2fd3c2cbd4a" exitCode=0 Jul 29 20:52:54.638962 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:54.638907 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp" event={"ID":"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c","Type":"ContainerDied","Data":"ae498a635e57ad2faf8c5dbc2f7ff78b87437eb0943081391daaa2fd3c2cbd4a"} Jul 29 20:52:54.641440 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:54.641416 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a5db510-214d-43a6-bfdd-a6d3ad45ca01" containerID="9fd2be577065bf23ad6fca7b78563b1a91fd67e94487ea747e2f35afa99b0e8a" exitCode=0 Jul 29 20:52:54.641531 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:54.641451 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9" event={"ID":"9a5db510-214d-43a6-bfdd-a6d3ad45ca01","Type":"ContainerDied","Data":"9fd2be577065bf23ad6fca7b78563b1a91fd67e94487ea747e2f35afa99b0e8a"} Jul 29 20:52:55.651781 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.651742 2562 generic.go:356] "Generic (PLEG): container finished" podID="c9eb5e4c-6906-441e-bfdf-a1f42e282365" containerID="eb6d097bea2598aa617044703955595f11176b42fa75c0284b8ede9d669ebe05" exitCode=0 Jul 29 20:52:55.652281 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.651831 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4" event={"ID":"c9eb5e4c-6906-441e-bfdf-a1f42e282365","Type":"ContainerDied","Data":"eb6d097bea2598aa617044703955595f11176b42fa75c0284b8ede9d669ebe05"} Jul 29 20:52:55.653541 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.653517 2562 generic.go:356] "Generic (PLEG): container finished" podID="84af0a1a-df6a-405a-9b11-0fcc1f090256" containerID="78591b48a2a94a60a3f9caf2c8de270c1b1f08a9856a238ec00f74654ba2c9a9" exitCode=0 Jul 29 20:52:55.653662 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.653602 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt" event={"ID":"84af0a1a-df6a-405a-9b11-0fcc1f090256","Type":"ContainerDied","Data":"78591b48a2a94a60a3f9caf2c8de270c1b1f08a9856a238ec00f74654ba2c9a9"} Jul 29 20:52:55.801785 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.801758 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9" Jul 29 20:52:55.841571 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.841541 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fp5b\" (UniqueName: \"kubernetes.io/projected/9a5db510-214d-43a6-bfdd-a6d3ad45ca01-kube-api-access-4fp5b\") pod \"9a5db510-214d-43a6-bfdd-a6d3ad45ca01\" (UID: \"9a5db510-214d-43a6-bfdd-a6d3ad45ca01\") " Jul 29 20:52:55.841694 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.841610 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a5db510-214d-43a6-bfdd-a6d3ad45ca01-util\") pod \"9a5db510-214d-43a6-bfdd-a6d3ad45ca01\" (UID: \"9a5db510-214d-43a6-bfdd-a6d3ad45ca01\") " Jul 29 20:52:55.841747 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.841726 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a5db510-214d-43a6-bfdd-a6d3ad45ca01-bundle\") pod \"9a5db510-214d-43a6-bfdd-a6d3ad45ca01\" (UID: \"9a5db510-214d-43a6-bfdd-a6d3ad45ca01\") " Jul 29 20:52:55.842192 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.842161 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5db510-214d-43a6-bfdd-a6d3ad45ca01-bundle" (OuterVolumeSpecName: "bundle") pod "9a5db510-214d-43a6-bfdd-a6d3ad45ca01" (UID: "9a5db510-214d-43a6-bfdd-a6d3ad45ca01"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:52:55.843723 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.843698 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5db510-214d-43a6-bfdd-a6d3ad45ca01-kube-api-access-4fp5b" (OuterVolumeSpecName: "kube-api-access-4fp5b") pod "9a5db510-214d-43a6-bfdd-a6d3ad45ca01" (UID: "9a5db510-214d-43a6-bfdd-a6d3ad45ca01"). InnerVolumeSpecName "kube-api-access-4fp5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:52:55.847187 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.847167 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5db510-214d-43a6-bfdd-a6d3ad45ca01-util" (OuterVolumeSpecName: "util") pod "9a5db510-214d-43a6-bfdd-a6d3ad45ca01" (UID: "9a5db510-214d-43a6-bfdd-a6d3ad45ca01"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:52:55.879194 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.879168 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp" Jul 29 20:52:55.943024 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.942903 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c-util\") pod \"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c\" (UID: \"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c\") " Jul 29 20:52:55.943206 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.943046 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c-bundle\") pod \"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c\" (UID: \"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c\") " Jul 29 20:52:55.943206 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.943112 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lpfl\" (UniqueName: \"kubernetes.io/projected/2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c-kube-api-access-9lpfl\") pod \"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c\" (UID: \"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c\") " Jul 29 20:52:55.943419 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.943400 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fp5b\" (UniqueName: \"kubernetes.io/projected/9a5db510-214d-43a6-bfdd-a6d3ad45ca01-kube-api-access-4fp5b\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:52:55.943483 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.943423 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a5db510-214d-43a6-bfdd-a6d3ad45ca01-util\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:52:55.943483 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.943438 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a5db510-214d-43a6-bfdd-a6d3ad45ca01-bundle\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:52:55.943550 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.943495 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c-bundle" (OuterVolumeSpecName: "bundle") pod "2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c" (UID: "2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:52:55.945211 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.945188 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c-kube-api-access-9lpfl" (OuterVolumeSpecName: "kube-api-access-9lpfl") pod "2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c" (UID: "2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c"). InnerVolumeSpecName "kube-api-access-9lpfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:52:55.948946 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:55.948909 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c-util" (OuterVolumeSpecName: "util") pod "2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c" (UID: "2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:52:56.044347 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.044305 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c-bundle\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:52:56.044347 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.044342 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lpfl\" (UniqueName: \"kubernetes.io/projected/2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c-kube-api-access-9lpfl\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:52:56.044347 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.044353 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c-util\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:52:56.661947 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.661904 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp" Jul 29 20:52:56.662369 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.661944 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp" event={"ID":"2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c","Type":"ContainerDied","Data":"ee7828eafe54455b6c94a92ccdf1a16da1583dafc3663eb96ca063bb1d0df248"} Jul 29 20:52:56.662369 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.661997 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee7828eafe54455b6c94a92ccdf1a16da1583dafc3663eb96ca063bb1d0df248" Jul 29 20:52:56.664108 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.664084 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9" Jul 29 20:52:56.664108 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.664095 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9" event={"ID":"9a5db510-214d-43a6-bfdd-a6d3ad45ca01","Type":"ContainerDied","Data":"9ff710471db1b43298580ca295c500c741101fe5a345d2b64e86e59b694b9351"} Jul 29 20:52:56.664281 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.664131 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ff710471db1b43298580ca295c500c741101fe5a345d2b64e86e59b694b9351" Jul 29 20:52:56.666521 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.666499 2562 generic.go:356] "Generic (PLEG): container finished" podID="84af0a1a-df6a-405a-9b11-0fcc1f090256" containerID="b93e44ee6b7dabec558cabd47e931b23828b40e0c66b9cafb96af96667efed3e" exitCode=0 Jul 29 20:52:56.666658 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.666571 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt" event={"ID":"84af0a1a-df6a-405a-9b11-0fcc1f090256","Type":"ContainerDied","Data":"b93e44ee6b7dabec558cabd47e931b23828b40e0c66b9cafb96af96667efed3e"} Jul 29 20:52:56.804080 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.804052 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4" Jul 29 20:52:56.851894 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.851854 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9eb5e4c-6906-441e-bfdf-a1f42e282365-util\") pod \"c9eb5e4c-6906-441e-bfdf-a1f42e282365\" (UID: \"c9eb5e4c-6906-441e-bfdf-a1f42e282365\") " Jul 29 20:52:56.851894 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.851895 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9eb5e4c-6906-441e-bfdf-a1f42e282365-bundle\") pod \"c9eb5e4c-6906-441e-bfdf-a1f42e282365\" (UID: \"c9eb5e4c-6906-441e-bfdf-a1f42e282365\") " Jul 29 20:52:56.852182 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.852045 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r65vq\" (UniqueName: \"kubernetes.io/projected/c9eb5e4c-6906-441e-bfdf-a1f42e282365-kube-api-access-r65vq\") pod \"c9eb5e4c-6906-441e-bfdf-a1f42e282365\" (UID: \"c9eb5e4c-6906-441e-bfdf-a1f42e282365\") " Jul 29 20:52:56.852492 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.852467 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9eb5e4c-6906-441e-bfdf-a1f42e282365-bundle" (OuterVolumeSpecName: "bundle") pod "c9eb5e4c-6906-441e-bfdf-a1f42e282365" (UID: "c9eb5e4c-6906-441e-bfdf-a1f42e282365"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:52:56.854141 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.854112 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9eb5e4c-6906-441e-bfdf-a1f42e282365-kube-api-access-r65vq" (OuterVolumeSpecName: "kube-api-access-r65vq") pod "c9eb5e4c-6906-441e-bfdf-a1f42e282365" (UID: "c9eb5e4c-6906-441e-bfdf-a1f42e282365"). InnerVolumeSpecName "kube-api-access-r65vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:52:56.857534 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.857510 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9eb5e4c-6906-441e-bfdf-a1f42e282365-util" (OuterVolumeSpecName: "util") pod "c9eb5e4c-6906-441e-bfdf-a1f42e282365" (UID: "c9eb5e4c-6906-441e-bfdf-a1f42e282365"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:52:56.953287 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.953192 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r65vq\" (UniqueName: \"kubernetes.io/projected/c9eb5e4c-6906-441e-bfdf-a1f42e282365-kube-api-access-r65vq\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:52:56.953287 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.953227 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9eb5e4c-6906-441e-bfdf-a1f42e282365-util\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:52:56.953287 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:56.953237 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9eb5e4c-6906-441e-bfdf-a1f42e282365-bundle\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:52:57.674373 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:57.674326 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4" event={"ID":"c9eb5e4c-6906-441e-bfdf-a1f42e282365","Type":"ContainerDied","Data":"d0e3eee3cf550ddcd13829729f53d38dce1629037bbc38b227fddc4b6ed6da1e"} Jul 29 20:52:57.674373 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:57.674356 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4" Jul 29 20:52:57.674891 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:57.674385 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0e3eee3cf550ddcd13829729f53d38dce1629037bbc38b227fddc4b6ed6da1e" Jul 29 20:52:57.815733 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:57.815671 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt" Jul 29 20:52:57.862331 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:57.862295 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdfqx\" (UniqueName: \"kubernetes.io/projected/84af0a1a-df6a-405a-9b11-0fcc1f090256-kube-api-access-sdfqx\") pod \"84af0a1a-df6a-405a-9b11-0fcc1f090256\" (UID: \"84af0a1a-df6a-405a-9b11-0fcc1f090256\") " Jul 29 20:52:57.862515 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:57.862415 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84af0a1a-df6a-405a-9b11-0fcc1f090256-util\") pod \"84af0a1a-df6a-405a-9b11-0fcc1f090256\" (UID: \"84af0a1a-df6a-405a-9b11-0fcc1f090256\") " Jul 29 20:52:57.862515 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:57.862460 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84af0a1a-df6a-405a-9b11-0fcc1f090256-bundle\") pod \"84af0a1a-df6a-405a-9b11-0fcc1f090256\" (UID: \"84af0a1a-df6a-405a-9b11-0fcc1f090256\") " Jul 29 20:52:57.863023 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:57.862994 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84af0a1a-df6a-405a-9b11-0fcc1f090256-bundle" (OuterVolumeSpecName: "bundle") pod "84af0a1a-df6a-405a-9b11-0fcc1f090256" (UID: "84af0a1a-df6a-405a-9b11-0fcc1f090256"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:52:57.864472 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:57.864448 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84af0a1a-df6a-405a-9b11-0fcc1f090256-kube-api-access-sdfqx" (OuterVolumeSpecName: "kube-api-access-sdfqx") pod "84af0a1a-df6a-405a-9b11-0fcc1f090256" (UID: "84af0a1a-df6a-405a-9b11-0fcc1f090256"). InnerVolumeSpecName "kube-api-access-sdfqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:52:57.867833 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:57.867809 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84af0a1a-df6a-405a-9b11-0fcc1f090256-util" (OuterVolumeSpecName: "util") pod "84af0a1a-df6a-405a-9b11-0fcc1f090256" (UID: "84af0a1a-df6a-405a-9b11-0fcc1f090256"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:52:57.963413 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:57.963372 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdfqx\" (UniqueName: \"kubernetes.io/projected/84af0a1a-df6a-405a-9b11-0fcc1f090256-kube-api-access-sdfqx\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:52:57.963413 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:57.963407 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84af0a1a-df6a-405a-9b11-0fcc1f090256-util\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:52:57.963413 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:57.963419 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84af0a1a-df6a-405a-9b11-0fcc1f090256-bundle\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:52:58.680913 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:58.680880 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt" Jul 29 20:52:58.681287 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:58.680889 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt" event={"ID":"84af0a1a-df6a-405a-9b11-0fcc1f090256","Type":"ContainerDied","Data":"ffcb2cd4f4ab2583142773023b68659cd47b435a9ab99078a0274c082e3d4782"} Jul 29 20:52:58.681287 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:52:58.680977 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffcb2cd4f4ab2583142773023b68659cd47b435a9ab99078a0274c082e3d4782" Jul 29 20:53:03.132415 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.132366 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f6fc64598-vdwfp"] Jul 29 20:53:03.133567 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133540 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5db510-214d-43a6-bfdd-a6d3ad45ca01" containerName="extract" Jul 29 20:53:03.133567 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133568 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5db510-214d-43a6-bfdd-a6d3ad45ca01" containerName="extract" Jul 29 20:53:03.133697 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133587 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9eb5e4c-6906-441e-bfdf-a1f42e282365" containerName="pull" Jul 29 20:53:03.133697 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133596 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9eb5e4c-6906-441e-bfdf-a1f42e282365" containerName="pull" Jul 29 20:53:03.133697 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133628 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84af0a1a-df6a-405a-9b11-0fcc1f090256" containerName="extract" Jul 29 20:53:03.133697 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133638 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="84af0a1a-df6a-405a-9b11-0fcc1f090256" containerName="extract" Jul 29 20:53:03.133697 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133668 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c" containerName="pull" Jul 29 20:53:03.133697 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133677 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c" containerName="pull" Jul 29 20:53:03.133697 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133693 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c" containerName="extract" Jul 29 20:53:03.133945 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133701 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c" containerName="extract" Jul 29 20:53:03.133945 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133711 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84af0a1a-df6a-405a-9b11-0fcc1f090256" containerName="util" Jul 29 20:53:03.133945 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133725 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="84af0a1a-df6a-405a-9b11-0fcc1f090256" containerName="util" Jul 29 20:53:03.133945 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133734 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9eb5e4c-6906-441e-bfdf-a1f42e282365" containerName="util" Jul 29 20:53:03.133945 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133742 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9eb5e4c-6906-441e-bfdf-a1f42e282365" containerName="util" Jul 29 20:53:03.133945 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133760 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9eb5e4c-6906-441e-bfdf-a1f42e282365" containerName="extract" Jul 29 20:53:03.133945 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133769 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9eb5e4c-6906-441e-bfdf-a1f42e282365" containerName="extract" Jul 29 20:53:03.133945 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133790 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5db510-214d-43a6-bfdd-a6d3ad45ca01" containerName="util" Jul 29 20:53:03.133945 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133798 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5db510-214d-43a6-bfdd-a6d3ad45ca01" containerName="util" Jul 29 20:53:03.133945 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133828 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c" containerName="util" Jul 29 20:53:03.133945 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133835 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c" containerName="util" Jul 29 20:53:03.133945 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133844 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84af0a1a-df6a-405a-9b11-0fcc1f090256" containerName="pull" Jul 29 20:53:03.133945 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133852 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="84af0a1a-df6a-405a-9b11-0fcc1f090256" containerName="pull" Jul 29 20:53:03.133945 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133872 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5db510-214d-43a6-bfdd-a6d3ad45ca01" containerName="pull" Jul 29 20:53:03.133945 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.133886 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5db510-214d-43a6-bfdd-a6d3ad45ca01" containerName="pull" Jul 29 20:53:03.134362 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.134078 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a5db510-214d-43a6-bfdd-a6d3ad45ca01" containerName="extract" Jul 29 20:53:03.134362 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.134099 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c" containerName="extract" Jul 29 20:53:03.134362 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.134108 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9eb5e4c-6906-441e-bfdf-a1f42e282365" containerName="extract" Jul 29 20:53:03.134362 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.134127 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="84af0a1a-df6a-405a-9b11-0fcc1f090256" containerName="extract" Jul 29 20:53:03.139060 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.139032 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.145256 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.145225 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f6fc64598-vdwfp"] Jul 29 20:53:03.215181 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.215137 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b01475-72a3-4c4e-b91a-20aa5c6cc175-trusted-ca-bundle\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.215355 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.215194 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73b01475-72a3-4c4e-b91a-20aa5c6cc175-console-oauth-config\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.215355 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.215272 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkckp\" (UniqueName: \"kubernetes.io/projected/73b01475-72a3-4c4e-b91a-20aa5c6cc175-kube-api-access-dkckp\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.215355 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.215304 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73b01475-72a3-4c4e-b91a-20aa5c6cc175-oauth-serving-cert\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.215355 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.215335 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73b01475-72a3-4c4e-b91a-20aa5c6cc175-console-serving-cert\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.215520 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.215420 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73b01475-72a3-4c4e-b91a-20aa5c6cc175-console-config\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.215520 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.215441 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73b01475-72a3-4c4e-b91a-20aa5c6cc175-service-ca\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.316633 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.316527 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkckp\" (UniqueName: \"kubernetes.io/projected/73b01475-72a3-4c4e-b91a-20aa5c6cc175-kube-api-access-dkckp\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.316794 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.316666 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73b01475-72a3-4c4e-b91a-20aa5c6cc175-oauth-serving-cert\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.316794 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.316709 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73b01475-72a3-4c4e-b91a-20aa5c6cc175-console-serving-cert\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.316794 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.316770 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73b01475-72a3-4c4e-b91a-20aa5c6cc175-console-config\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.316794 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.316786 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73b01475-72a3-4c4e-b91a-20aa5c6cc175-service-ca\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.317038 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.316823 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b01475-72a3-4c4e-b91a-20aa5c6cc175-trusted-ca-bundle\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.317038 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.316867 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73b01475-72a3-4c4e-b91a-20aa5c6cc175-console-oauth-config\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.317422 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.317394 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73b01475-72a3-4c4e-b91a-20aa5c6cc175-oauth-serving-cert\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.317536 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.317510 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73b01475-72a3-4c4e-b91a-20aa5c6cc175-console-config\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.317623 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.317600 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73b01475-72a3-4c4e-b91a-20aa5c6cc175-service-ca\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.317768 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.317743 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b01475-72a3-4c4e-b91a-20aa5c6cc175-trusted-ca-bundle\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.319299 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.319275 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73b01475-72a3-4c4e-b91a-20aa5c6cc175-console-serving-cert\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.319419 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.319396 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73b01475-72a3-4c4e-b91a-20aa5c6cc175-console-oauth-config\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.325042 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.325017 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkckp\" (UniqueName: \"kubernetes.io/projected/73b01475-72a3-4c4e-b91a-20aa5c6cc175-kube-api-access-dkckp\") pod \"console-6f6fc64598-vdwfp\" (UID: \"73b01475-72a3-4c4e-b91a-20aa5c6cc175\") " pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.453984 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.453864 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:03.596149 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.596119 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f6fc64598-vdwfp"] Jul 29 20:53:03.597786 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:53:03.597757 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b01475_72a3_4c4e_b91a_20aa5c6cc175.slice/crio-307adf72214df7ec13d3df11eecc75be403a5c46c5ac6bcfecc4f6bc839ec344 WatchSource:0}: Error finding container 307adf72214df7ec13d3df11eecc75be403a5c46c5ac6bcfecc4f6bc839ec344: Status 404 returned error can't find the container with id 307adf72214df7ec13d3df11eecc75be403a5c46c5ac6bcfecc4f6bc839ec344 Jul 29 20:53:03.709359 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.709258 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f6fc64598-vdwfp" event={"ID":"73b01475-72a3-4c4e-b91a-20aa5c6cc175","Type":"ContainerStarted","Data":"d9c8666245fa1337777225041eb38af0d4ddfda1f7bac9abd32f589986f64c4e"} Jul 29 20:53:03.709359 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.709330 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f6fc64598-vdwfp" event={"ID":"73b01475-72a3-4c4e-b91a-20aa5c6cc175","Type":"ContainerStarted","Data":"307adf72214df7ec13d3df11eecc75be403a5c46c5ac6bcfecc4f6bc839ec344"} Jul 29 20:53:03.725904 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:03.725841 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f6fc64598-vdwfp" podStartSLOduration=0.725823592 podStartE2EDuration="725.823592ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:53:03.724441258 +0000 UTC m=+400.595544734" watchObservedRunningTime="2026-07-29 20:53:03.725823592 +0000 UTC m=+400.596927060" Jul 29 20:53:07.481424 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:07.481379 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fzznx"] Jul 29 20:53:07.485119 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:07.485093 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fzznx" Jul 29 20:53:07.486906 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:07.486882 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 29 20:53:07.487017 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:07.486955 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7xpj8\"" Jul 29 20:53:07.494063 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:07.494038 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fzznx"] Jul 29 20:53:07.555754 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:07.555721 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxgf7\" (UniqueName: \"kubernetes.io/projected/38178c0f-2d1b-435b-8c09-944bc9f8152d-kube-api-access-fxgf7\") pod \"dns-operator-controller-manager-864dd78c4c-fzznx\" (UID: \"38178c0f-2d1b-435b-8c09-944bc9f8152d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fzznx" Jul 29 20:53:07.657204 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:07.657162 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxgf7\" (UniqueName: \"kubernetes.io/projected/38178c0f-2d1b-435b-8c09-944bc9f8152d-kube-api-access-fxgf7\") pod \"dns-operator-controller-manager-864dd78c4c-fzznx\" (UID: \"38178c0f-2d1b-435b-8c09-944bc9f8152d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fzznx" Jul 29 20:53:07.670836 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:07.670798 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxgf7\" (UniqueName: \"kubernetes.io/projected/38178c0f-2d1b-435b-8c09-944bc9f8152d-kube-api-access-fxgf7\") pod \"dns-operator-controller-manager-864dd78c4c-fzznx\" (UID: \"38178c0f-2d1b-435b-8c09-944bc9f8152d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fzznx" Jul 29 20:53:07.796761 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:07.796662 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fzznx" Jul 29 20:53:07.931702 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:07.931663 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fzznx"] Jul 29 20:53:07.932993 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:53:07.932964 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38178c0f_2d1b_435b_8c09_944bc9f8152d.slice/crio-99a41991a8ba81914fd66e23e2aba67a8680d2f748ce392cd91560eddb5856b4 WatchSource:0}: Error finding container 99a41991a8ba81914fd66e23e2aba67a8680d2f748ce392cd91560eddb5856b4: Status 404 returned error can't find the container with id 99a41991a8ba81914fd66e23e2aba67a8680d2f748ce392cd91560eddb5856b4 Jul 29 20:53:08.739355 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:08.739317 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fzznx" event={"ID":"38178c0f-2d1b-435b-8c09-944bc9f8152d","Type":"ContainerStarted","Data":"99a41991a8ba81914fd66e23e2aba67a8680d2f748ce392cd91560eddb5856b4"} Jul 29 20:53:10.752545 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:10.752503 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fzznx" event={"ID":"38178c0f-2d1b-435b-8c09-944bc9f8152d","Type":"ContainerStarted","Data":"837c1a8d0440899eda697577bc98cac20433830cffb40e3b0a50ccb79f3159dc"} Jul 29 20:53:10.752979 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:10.752693 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fzznx" Jul 29 20:53:10.768014 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:10.767952 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fzznx" podStartSLOduration=1.613068791 podStartE2EDuration="3.767905097s" totalImagesPullingTime="2.154836306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:53:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:53:10.767352161 +0000 UTC m=+407.638455625" watchObservedRunningTime="2026-07-29 20:53:10.767905097 +0000 UTC m=+407.639008568" Jul 29 20:53:12.845062 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:12.845020 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn"] Jul 29 20:53:12.848872 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:12.848848 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:12.851739 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:12.851718 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mbgcs\"" Jul 29 20:53:12.872067 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:12.872036 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn"] Jul 29 20:53:13.008105 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:13.008063 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjw9w\" (UniqueName: \"kubernetes.io/projected/eec96a2e-a8ba-4705-81a4-ce3b64da9e7a-kube-api-access-xjw9w\") pod \"kuadrant-operator-controller-manager-8684c9ff49-flzsn\" (UID: \"eec96a2e-a8ba-4705-81a4-ce3b64da9e7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:13.008289 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:13.008214 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eec96a2e-a8ba-4705-81a4-ce3b64da9e7a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-flzsn\" (UID: \"eec96a2e-a8ba-4705-81a4-ce3b64da9e7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:13.109242 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:13.109133 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjw9w\" (UniqueName: \"kubernetes.io/projected/eec96a2e-a8ba-4705-81a4-ce3b64da9e7a-kube-api-access-xjw9w\") pod \"kuadrant-operator-controller-manager-8684c9ff49-flzsn\" (UID: \"eec96a2e-a8ba-4705-81a4-ce3b64da9e7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:13.109242 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:13.109244 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eec96a2e-a8ba-4705-81a4-ce3b64da9e7a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-flzsn\" (UID: \"eec96a2e-a8ba-4705-81a4-ce3b64da9e7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:13.109614 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:13.109597 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eec96a2e-a8ba-4705-81a4-ce3b64da9e7a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-flzsn\" (UID: \"eec96a2e-a8ba-4705-81a4-ce3b64da9e7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:13.121290 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:13.121261 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjw9w\" (UniqueName: \"kubernetes.io/projected/eec96a2e-a8ba-4705-81a4-ce3b64da9e7a-kube-api-access-xjw9w\") pod \"kuadrant-operator-controller-manager-8684c9ff49-flzsn\" (UID: \"eec96a2e-a8ba-4705-81a4-ce3b64da9e7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:13.161122 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:13.161050 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:13.317173 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:13.317145 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn"] Jul 29 20:53:13.319086 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:53:13.319058 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec96a2e_a8ba_4705_81a4_ce3b64da9e7a.slice/crio-20b81986b93db6b7eb2b5f668d9aee767f685791f023a8c5f265c7ab57a9982b WatchSource:0}: Error finding container 20b81986b93db6b7eb2b5f668d9aee767f685791f023a8c5f265c7ab57a9982b: Status 404 returned error can't find the container with id 20b81986b93db6b7eb2b5f668d9aee767f685791f023a8c5f265c7ab57a9982b Jul 29 20:53:13.454191 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:13.454087 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:13.454390 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:13.454367 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:13.459118 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:13.459098 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:13.772430 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:13.772322 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" event={"ID":"eec96a2e-a8ba-4705-81a4-ce3b64da9e7a","Type":"ContainerStarted","Data":"20b81986b93db6b7eb2b5f668d9aee767f685791f023a8c5f265c7ab57a9982b"} Jul 29 20:53:13.776502 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:13.776475 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f6fc64598-vdwfp" Jul 29 20:53:13.826186 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:13.826148 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bc6c7f784-bhnlb"] Jul 29 20:53:19.075862 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:19.075832 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb"] Jul 29 20:53:19.080681 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:19.080661 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" Jul 29 20:53:19.083097 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:19.083069 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-s286g\"" Jul 29 20:53:19.091710 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:19.091683 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb"] Jul 29 20:53:19.275570 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:19.275527 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc2nk\" (UniqueName: \"kubernetes.io/projected/7f70c7d2-fa90-46c6-9568-ce2bf1079b85-kube-api-access-tc2nk\") pod \"limitador-operator-controller-manager-859d7b78bd-v29fb\" (UID: \"7f70c7d2-fa90-46c6-9568-ce2bf1079b85\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" Jul 29 20:53:19.377055 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:19.377009 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc2nk\" (UniqueName: \"kubernetes.io/projected/7f70c7d2-fa90-46c6-9568-ce2bf1079b85-kube-api-access-tc2nk\") pod \"limitador-operator-controller-manager-859d7b78bd-v29fb\" (UID: \"7f70c7d2-fa90-46c6-9568-ce2bf1079b85\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" Jul 29 20:53:19.384554 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:19.384523 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc2nk\" (UniqueName: \"kubernetes.io/projected/7f70c7d2-fa90-46c6-9568-ce2bf1079b85-kube-api-access-tc2nk\") pod \"limitador-operator-controller-manager-859d7b78bd-v29fb\" (UID: \"7f70c7d2-fa90-46c6-9568-ce2bf1079b85\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" Jul 29 20:53:19.408510 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:19.408474 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" Jul 29 20:53:19.559094 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:19.559063 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb"] Jul 29 20:53:19.559343 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:53:19.559317 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f70c7d2_fa90_46c6_9568_ce2bf1079b85.slice/crio-219542133bbbcc807dd12a4618f2ddf1e8584fc6635d9450442d3e2ce1c7d1d0 WatchSource:0}: Error finding container 219542133bbbcc807dd12a4618f2ddf1e8584fc6635d9450442d3e2ce1c7d1d0: Status 404 returned error can't find the container with id 219542133bbbcc807dd12a4618f2ddf1e8584fc6635d9450442d3e2ce1c7d1d0 Jul 29 20:53:19.824018 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:19.823974 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" event={"ID":"7f70c7d2-fa90-46c6-9568-ce2bf1079b85","Type":"ContainerStarted","Data":"219542133bbbcc807dd12a4618f2ddf1e8584fc6635d9450442d3e2ce1c7d1d0"} Jul 29 20:53:19.825616 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:19.825591 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" event={"ID":"eec96a2e-a8ba-4705-81a4-ce3b64da9e7a","Type":"ContainerStarted","Data":"c7deba12ad6a7984a9272eab38a110f0e5b992c8179df6a0dfd94a79cfbea314"} Jul 29 20:53:19.825740 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:19.825705 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:19.852559 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:19.852504 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" podStartSLOduration=2.1292007809999998 podStartE2EDuration="7.852486126s" totalImagesPullingTime="5.723285345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:53:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:53:19.846530957 +0000 UTC m=+416.717634447" watchObservedRunningTime="2026-07-29 20:53:19.852486126 +0000 UTC m=+416.723589592" Jul 29 20:53:21.761152 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:21.761122 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fzznx" Jul 29 20:53:21.838747 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:21.838714 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" event={"ID":"7f70c7d2-fa90-46c6-9568-ce2bf1079b85","Type":"ContainerStarted","Data":"0c827563f1a3d72ed61f20a2e3227702e5ea9845e07a4f48168c570aadee13d9"} Jul 29 20:53:21.838932 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:21.838811 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" Jul 29 20:53:21.857181 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:21.857125 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" podStartSLOduration=1.227989109 podStartE2EDuration="2.857108017s" totalImagesPullingTime="1.629118908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:53:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:53:21.855843839 +0000 UTC m=+418.726947309" watchObservedRunningTime="2026-07-29 20:53:21.857108017 +0000 UTC m=+418.728211484" Jul 29 20:53:30.834196 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:30.834160 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:32.473342 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.473303 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn"] Jul 29 20:53:32.473817 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.473493 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" podUID="eec96a2e-a8ba-4705-81a4-ce3b64da9e7a" containerName="manager" containerID="cri-o://c7deba12ad6a7984a9272eab38a110f0e5b992c8179df6a0dfd94a79cfbea314" gracePeriod=2 Jul 29 20:53:32.482891 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.482853 2562 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"eec96a2e-a8ba-4705-81a4-ce3b64da9e7a\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-flzsn\": pods \"kuadrant-operator-controller-manager-8684c9ff49-flzsn\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:32.483518 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.483466 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn"] Jul 29 20:53:32.493426 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.493396 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb"] Jul 29 20:53:32.493688 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.493596 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" podUID="7f70c7d2-fa90-46c6-9568-ce2bf1079b85" containerName="manager" containerID="cri-o://0c827563f1a3d72ed61f20a2e3227702e5ea9845e07a4f48168c570aadee13d9" gracePeriod=2 Jul 29 20:53:32.495741 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.495716 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" Jul 29 20:53:32.500464 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.500272 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4"] Jul 29 20:53:32.500801 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.500778 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eec96a2e-a8ba-4705-81a4-ce3b64da9e7a" containerName="manager" Jul 29 20:53:32.500901 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.500804 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="eec96a2e-a8ba-4705-81a4-ce3b64da9e7a" containerName="manager" Jul 29 20:53:32.500901 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.500866 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="eec96a2e-a8ba-4705-81a4-ce3b64da9e7a" containerName="manager" Jul 29 20:53:32.504168 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.504146 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4" Jul 29 20:53:32.508172 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.507907 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb"] Jul 29 20:53:32.511158 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.511125 2562 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"7f70c7d2-fa90-46c6-9568-ce2bf1079b85\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-v29fb\": pods \"limitador-operator-controller-manager-859d7b78bd-v29fb\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" Jul 29 20:53:32.512773 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.512743 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-flzsn\" is forbidden: User \"system:node:ip-10-0-137-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-216.ec2.internal' and this object" podUID="eec96a2e-a8ba-4705-81a4-ce3b64da9e7a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:32.516596 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.516560 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-v29fb\" is forbidden: User \"system:node:ip-10-0-137-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-216.ec2.internal' and this object" podUID="7f70c7d2-fa90-46c6-9568-ce2bf1079b85" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" Jul 29 20:53:32.518677 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.518646 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-flzsn\" is forbidden: User \"system:node:ip-10-0-137-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-216.ec2.internal' and this object" podUID="eec96a2e-a8ba-4705-81a4-ce3b64da9e7a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:32.518929 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.518887 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4"] Jul 29 20:53:32.579205 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.579173 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch"] Jul 29 20:53:32.579736 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.579715 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f70c7d2-fa90-46c6-9568-ce2bf1079b85" containerName="manager" Jul 29 20:53:32.579736 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.579736 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f70c7d2-fa90-46c6-9568-ce2bf1079b85" containerName="manager" Jul 29 20:53:32.579977 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.579847 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f70c7d2-fa90-46c6-9568-ce2bf1079b85" containerName="manager" Jul 29 20:53:32.583032 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.583014 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch" Jul 29 20:53:32.595393 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.595363 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch"] Jul 29 20:53:32.598053 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.598025 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/acea6479-8752-4949-ac64-1d40a96e87ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fqsm4\" (UID: \"acea6479-8752-4949-ac64-1d40a96e87ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4" Jul 29 20:53:32.598360 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.598322 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9mfn\" (UniqueName: \"kubernetes.io/projected/acea6479-8752-4949-ac64-1d40a96e87ab-kube-api-access-s9mfn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fqsm4\" (UID: \"acea6479-8752-4949-ac64-1d40a96e87ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4" Jul 29 20:53:32.608430 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.608391 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-v29fb\" is forbidden: User \"system:node:ip-10-0-137-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-216.ec2.internal' and this object" podUID="7f70c7d2-fa90-46c6-9568-ce2bf1079b85" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" Jul 29 20:53:32.610075 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.610045 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-flzsn\" is forbidden: User \"system:node:ip-10-0-137-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-216.ec2.internal' and this object" podUID="eec96a2e-a8ba-4705-81a4-ce3b64da9e7a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:32.699540 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.699499 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee5e5207-ca36-433e-9d3d-0fb90c066122-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-xfmch\" (UID: \"ee5e5207-ca36-433e-9d3d-0fb90c066122\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch" Jul 29 20:53:32.699709 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.699570 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9mfn\" (UniqueName: \"kubernetes.io/projected/acea6479-8752-4949-ac64-1d40a96e87ab-kube-api-access-s9mfn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fqsm4\" (UID: \"acea6479-8752-4949-ac64-1d40a96e87ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4" Jul 29 20:53:32.699709 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.699673 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm9pl\" (UniqueName: \"kubernetes.io/projected/ee5e5207-ca36-433e-9d3d-0fb90c066122-kube-api-access-fm9pl\") pod \"kuadrant-operator-controller-manager-86878c769f-xfmch\" (UID: \"ee5e5207-ca36-433e-9d3d-0fb90c066122\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch" Jul 29 20:53:32.699796 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.699769 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/acea6479-8752-4949-ac64-1d40a96e87ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fqsm4\" (UID: \"acea6479-8752-4949-ac64-1d40a96e87ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4" Jul 29 20:53:32.700232 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.700209 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/acea6479-8752-4949-ac64-1d40a96e87ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fqsm4\" (UID: \"acea6479-8752-4949-ac64-1d40a96e87ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4" Jul 29 20:53:32.710444 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.710412 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9mfn\" (UniqueName: \"kubernetes.io/projected/acea6479-8752-4949-ac64-1d40a96e87ab-kube-api-access-s9mfn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fqsm4\" (UID: \"acea6479-8752-4949-ac64-1d40a96e87ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4" Jul 29 20:53:32.752800 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.752772 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:32.754670 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.754639 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-v29fb\" is forbidden: User \"system:node:ip-10-0-137-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-216.ec2.internal' and this object" podUID="7f70c7d2-fa90-46c6-9568-ce2bf1079b85" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" Jul 29 20:53:32.755192 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.755173 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" Jul 29 20:53:32.756449 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.756427 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-flzsn\" is forbidden: User \"system:node:ip-10-0-137-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-216.ec2.internal' and this object" podUID="eec96a2e-a8ba-4705-81a4-ce3b64da9e7a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:32.758454 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.758430 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-v29fb\" is forbidden: User \"system:node:ip-10-0-137-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-216.ec2.internal' and this object" podUID="7f70c7d2-fa90-46c6-9568-ce2bf1079b85" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" Jul 29 20:53:32.759718 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.759696 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-flzsn\" is forbidden: User \"system:node:ip-10-0-137-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-216.ec2.internal' and this object" podUID="eec96a2e-a8ba-4705-81a4-ce3b64da9e7a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:32.800365 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.800327 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eec96a2e-a8ba-4705-81a4-ce3b64da9e7a-extensions-socket-volume\") pod \"eec96a2e-a8ba-4705-81a4-ce3b64da9e7a\" (UID: \"eec96a2e-a8ba-4705-81a4-ce3b64da9e7a\") " Jul 29 20:53:32.800512 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.800365 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tc2nk\" (UniqueName: \"kubernetes.io/projected/7f70c7d2-fa90-46c6-9568-ce2bf1079b85-kube-api-access-tc2nk\") pod \"7f70c7d2-fa90-46c6-9568-ce2bf1079b85\" (UID: \"7f70c7d2-fa90-46c6-9568-ce2bf1079b85\") " Jul 29 20:53:32.800512 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.800400 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjw9w\" (UniqueName: \"kubernetes.io/projected/eec96a2e-a8ba-4705-81a4-ce3b64da9e7a-kube-api-access-xjw9w\") pod \"eec96a2e-a8ba-4705-81a4-ce3b64da9e7a\" (UID: \"eec96a2e-a8ba-4705-81a4-ce3b64da9e7a\") " Jul 29 20:53:32.800606 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.800584 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee5e5207-ca36-433e-9d3d-0fb90c066122-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-xfmch\" (UID: \"ee5e5207-ca36-433e-9d3d-0fb90c066122\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch" Jul 29 20:53:32.800672 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.800658 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm9pl\" (UniqueName: \"kubernetes.io/projected/ee5e5207-ca36-433e-9d3d-0fb90c066122-kube-api-access-fm9pl\") pod \"kuadrant-operator-controller-manager-86878c769f-xfmch\" (UID: \"ee5e5207-ca36-433e-9d3d-0fb90c066122\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch" Jul 29 20:53:32.800836 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.800800 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec96a2e-a8ba-4705-81a4-ce3b64da9e7a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "eec96a2e-a8ba-4705-81a4-ce3b64da9e7a" (UID: "eec96a2e-a8ba-4705-81a4-ce3b64da9e7a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:53:32.801014 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.800974 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee5e5207-ca36-433e-9d3d-0fb90c066122-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-xfmch\" (UID: \"ee5e5207-ca36-433e-9d3d-0fb90c066122\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch" Jul 29 20:53:32.802671 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.802644 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec96a2e-a8ba-4705-81a4-ce3b64da9e7a-kube-api-access-xjw9w" (OuterVolumeSpecName: "kube-api-access-xjw9w") pod "eec96a2e-a8ba-4705-81a4-ce3b64da9e7a" (UID: "eec96a2e-a8ba-4705-81a4-ce3b64da9e7a"). InnerVolumeSpecName "kube-api-access-xjw9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:53:32.802769 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.802703 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f70c7d2-fa90-46c6-9568-ce2bf1079b85-kube-api-access-tc2nk" (OuterVolumeSpecName: "kube-api-access-tc2nk") pod "7f70c7d2-fa90-46c6-9568-ce2bf1079b85" (UID: "7f70c7d2-fa90-46c6-9568-ce2bf1079b85"). InnerVolumeSpecName "kube-api-access-tc2nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:53:32.811796 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.811769 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm9pl\" (UniqueName: \"kubernetes.io/projected/ee5e5207-ca36-433e-9d3d-0fb90c066122-kube-api-access-fm9pl\") pod \"kuadrant-operator-controller-manager-86878c769f-xfmch\" (UID: \"ee5e5207-ca36-433e-9d3d-0fb90c066122\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch" Jul 29 20:53:32.901607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.901571 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eec96a2e-a8ba-4705-81a4-ce3b64da9e7a-extensions-socket-volume\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:53:32.901607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.901602 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc2nk\" (UniqueName: \"kubernetes.io/projected/7f70c7d2-fa90-46c6-9568-ce2bf1079b85-kube-api-access-tc2nk\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:53:32.901607 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.901616 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjw9w\" (UniqueName: \"kubernetes.io/projected/eec96a2e-a8ba-4705-81a4-ce3b64da9e7a-kube-api-access-xjw9w\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:53:32.903335 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.903304 2562 generic.go:356] "Generic (PLEG): container finished" podID="7f70c7d2-fa90-46c6-9568-ce2bf1079b85" containerID="0c827563f1a3d72ed61f20a2e3227702e5ea9845e07a4f48168c570aadee13d9" exitCode=0 Jul 29 20:53:32.903490 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.903354 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" Jul 29 20:53:32.903490 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.903416 2562 scope.go:117] "RemoveContainer" containerID="0c827563f1a3d72ed61f20a2e3227702e5ea9845e07a4f48168c570aadee13d9" Jul 29 20:53:32.904910 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.904885 2562 generic.go:356] "Generic (PLEG): container finished" podID="eec96a2e-a8ba-4705-81a4-ce3b64da9e7a" containerID="c7deba12ad6a7984a9272eab38a110f0e5b992c8179df6a0dfd94a79cfbea314" exitCode=0 Jul 29 20:53:32.905017 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.904957 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:32.905334 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.905302 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-flzsn\" is forbidden: User \"system:node:ip-10-0-137-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-216.ec2.internal' and this object" podUID="eec96a2e-a8ba-4705-81a4-ce3b64da9e7a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:32.906593 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.906572 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-v29fb\" is forbidden: User \"system:node:ip-10-0-137-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-216.ec2.internal' and this object" podUID="7f70c7d2-fa90-46c6-9568-ce2bf1079b85" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" Jul 29 20:53:32.907934 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.907894 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-v29fb\" is forbidden: User \"system:node:ip-10-0-137-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-216.ec2.internal' and this object" podUID="7f70c7d2-fa90-46c6-9568-ce2bf1079b85" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" Jul 29 20:53:32.910294 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.910271 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-flzsn\" is forbidden: User \"system:node:ip-10-0-137-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-216.ec2.internal' and this object" podUID="eec96a2e-a8ba-4705-81a4-ce3b64da9e7a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:32.914779 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.914756 2562 scope.go:117] "RemoveContainer" containerID="0c827563f1a3d72ed61f20a2e3227702e5ea9845e07a4f48168c570aadee13d9" Jul 29 20:53:32.915069 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.915041 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-flzsn\" is forbidden: User \"system:node:ip-10-0-137-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-216.ec2.internal' and this object" podUID="eec96a2e-a8ba-4705-81a4-ce3b64da9e7a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:32.915170 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.915137 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c827563f1a3d72ed61f20a2e3227702e5ea9845e07a4f48168c570aadee13d9\": container with ID starting with 0c827563f1a3d72ed61f20a2e3227702e5ea9845e07a4f48168c570aadee13d9 not found: ID does not exist" containerID="0c827563f1a3d72ed61f20a2e3227702e5ea9845e07a4f48168c570aadee13d9" Jul 29 20:53:32.915225 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.915165 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c827563f1a3d72ed61f20a2e3227702e5ea9845e07a4f48168c570aadee13d9"} err="failed to get container status \"0c827563f1a3d72ed61f20a2e3227702e5ea9845e07a4f48168c570aadee13d9\": rpc error: code = NotFound desc = could not find container \"0c827563f1a3d72ed61f20a2e3227702e5ea9845e07a4f48168c570aadee13d9\": container with ID starting with 0c827563f1a3d72ed61f20a2e3227702e5ea9845e07a4f48168c570aadee13d9 not found: ID does not exist" Jul 29 20:53:32.915225 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.915200 2562 scope.go:117] "RemoveContainer" containerID="c7deba12ad6a7984a9272eab38a110f0e5b992c8179df6a0dfd94a79cfbea314" Jul 29 20:53:32.916301 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.916289 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4" Jul 29 20:53:32.916907 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.916886 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-v29fb\" is forbidden: User \"system:node:ip-10-0-137-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-216.ec2.internal' and this object" podUID="7f70c7d2-fa90-46c6-9568-ce2bf1079b85" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" Jul 29 20:53:32.920055 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.920031 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-v29fb\" is forbidden: User \"system:node:ip-10-0-137-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-216.ec2.internal' and this object" podUID="7f70c7d2-fa90-46c6-9568-ce2bf1079b85" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" Jul 29 20:53:32.921654 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.921633 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-flzsn\" is forbidden: User \"system:node:ip-10-0-137-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-216.ec2.internal' and this object" podUID="eec96a2e-a8ba-4705-81a4-ce3b64da9e7a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:32.924517 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.924503 2562 scope.go:117] "RemoveContainer" containerID="c7deba12ad6a7984a9272eab38a110f0e5b992c8179df6a0dfd94a79cfbea314" Jul 29 20:53:32.924776 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.924759 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch" Jul 29 20:53:32.924843 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:32.924777 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7deba12ad6a7984a9272eab38a110f0e5b992c8179df6a0dfd94a79cfbea314\": container with ID starting with c7deba12ad6a7984a9272eab38a110f0e5b992c8179df6a0dfd94a79cfbea314 not found: ID does not exist" containerID="c7deba12ad6a7984a9272eab38a110f0e5b992c8179df6a0dfd94a79cfbea314" Jul 29 20:53:32.924843 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:32.924807 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7deba12ad6a7984a9272eab38a110f0e5b992c8179df6a0dfd94a79cfbea314"} err="failed to get container status \"c7deba12ad6a7984a9272eab38a110f0e5b992c8179df6a0dfd94a79cfbea314\": rpc error: code = NotFound desc = could not find container \"c7deba12ad6a7984a9272eab38a110f0e5b992c8179df6a0dfd94a79cfbea314\": container with ID starting with c7deba12ad6a7984a9272eab38a110f0e5b992c8179df6a0dfd94a79cfbea314 not found: ID does not exist" Jul 29 20:53:33.088132 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:33.088101 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4"] Jul 29 20:53:33.088869 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:53:33.088843 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacea6479_8752_4949_ac64_1d40a96e87ab.slice/crio-725f6ca86d43133384fcbff04faa789909fec64d36c56320695f6c51d5bf6225 WatchSource:0}: Error finding container 725f6ca86d43133384fcbff04faa789909fec64d36c56320695f6c51d5bf6225: Status 404 returned error can't find the container with id 725f6ca86d43133384fcbff04faa789909fec64d36c56320695f6c51d5bf6225 Jul 29 20:53:33.098496 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:33.098469 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch"] Jul 29 20:53:33.103785 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:53:33.103757 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5e5207_ca36_433e_9d3d_0fb90c066122.slice/crio-e4bfd9b026a50bd5150b630f886c9d2fbc52f6683db314f5730bb74a0a3e570c WatchSource:0}: Error finding container e4bfd9b026a50bd5150b630f886c9d2fbc52f6683db314f5730bb74a0a3e570c: Status 404 returned error can't find the container with id e4bfd9b026a50bd5150b630f886c9d2fbc52f6683db314f5730bb74a0a3e570c Jul 29 20:53:33.558954 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:33.558091 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-v29fb\" is forbidden: User \"system:node:ip-10-0-137-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-216.ec2.internal' and this object" podUID="7f70c7d2-fa90-46c6-9568-ce2bf1079b85" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v29fb" Jul 29 20:53:33.559638 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:33.559602 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-flzsn\" is forbidden: User \"system:node:ip-10-0-137-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-216.ec2.internal' and this object" podUID="eec96a2e-a8ba-4705-81a4-ce3b64da9e7a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-flzsn" Jul 29 20:53:33.564807 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:33.564781 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f70c7d2-fa90-46c6-9568-ce2bf1079b85" path="/var/lib/kubelet/pods/7f70c7d2-fa90-46c6-9568-ce2bf1079b85/volumes" Jul 29 20:53:33.565170 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:33.565154 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eec96a2e-a8ba-4705-81a4-ce3b64da9e7a" path="/var/lib/kubelet/pods/eec96a2e-a8ba-4705-81a4-ce3b64da9e7a/volumes" Jul 29 20:53:33.915013 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:33.914976 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4" event={"ID":"acea6479-8752-4949-ac64-1d40a96e87ab","Type":"ContainerStarted","Data":"5ef2b0d8152761cd2f02b041d4a10f5396461b25ad69eec5de827e18dd602c00"} Jul 29 20:53:33.915237 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:33.915047 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4" Jul 29 20:53:33.915237 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:33.915080 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4" event={"ID":"acea6479-8752-4949-ac64-1d40a96e87ab","Type":"ContainerStarted","Data":"725f6ca86d43133384fcbff04faa789909fec64d36c56320695f6c51d5bf6225"} Jul 29 20:53:33.917648 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:33.917622 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch" event={"ID":"ee5e5207-ca36-433e-9d3d-0fb90c066122","Type":"ContainerStarted","Data":"a67bd65284ee12d88108f22575341b20b5b5df50c84c81eafb82b555beae0c53"} Jul 29 20:53:33.917764 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:33.917662 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch" event={"ID":"ee5e5207-ca36-433e-9d3d-0fb90c066122","Type":"ContainerStarted","Data":"e4bfd9b026a50bd5150b630f886c9d2fbc52f6683db314f5730bb74a0a3e570c"} Jul 29 20:53:33.917859 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:33.917772 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch" Jul 29 20:53:33.938796 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:33.938724 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4" podStartSLOduration=1.938704289 podStartE2EDuration="1.938704289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:53:33.938183093 +0000 UTC m=+430.809286581" watchObservedRunningTime="2026-07-29 20:53:33.938704289 +0000 UTC m=+430.809807758" Jul 29 20:53:33.961904 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:33.961843 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch" podStartSLOduration=1.9618265030000002 podStartE2EDuration="1.961826503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:53:33.960459325 +0000 UTC m=+430.831562791" watchObservedRunningTime="2026-07-29 20:53:33.961826503 +0000 UTC m=+430.832930023" Jul 29 20:53:38.851904 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:38.851863 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bc6c7f784-bhnlb" podUID="5009db96-8df4-4785-858a-0c2ab892c94a" containerName="console" containerID="cri-o://80adcf83691aad1229f47ed364721cf9fd92c059acf443a21057aed7e1a872be" gracePeriod=15 Jul 29 20:53:39.104892 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.104826 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:53:39.165034 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.164997 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5009db96-8df4-4785-858a-0c2ab892c94a-console-oauth-config\") pod \"5009db96-8df4-4785-858a-0c2ab892c94a\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " Jul 29 20:53:39.165228 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.165050 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjz8w\" (UniqueName: \"kubernetes.io/projected/5009db96-8df4-4785-858a-0c2ab892c94a-kube-api-access-sjz8w\") pod \"5009db96-8df4-4785-858a-0c2ab892c94a\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " Jul 29 20:53:39.165228 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.165082 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-console-config\") pod \"5009db96-8df4-4785-858a-0c2ab892c94a\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " Jul 29 20:53:39.165228 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.165127 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5009db96-8df4-4785-858a-0c2ab892c94a-console-serving-cert\") pod \"5009db96-8df4-4785-858a-0c2ab892c94a\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " Jul 29 20:53:39.165378 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.165274 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-oauth-serving-cert\") pod \"5009db96-8df4-4785-858a-0c2ab892c94a\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " Jul 29 20:53:39.165378 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.165366 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-service-ca\") pod \"5009db96-8df4-4785-858a-0c2ab892c94a\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " Jul 29 20:53:39.165490 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.165410 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-trusted-ca-bundle\") pod \"5009db96-8df4-4785-858a-0c2ab892c94a\" (UID: \"5009db96-8df4-4785-858a-0c2ab892c94a\") " Jul 29 20:53:39.165697 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.165556 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-console-config" (OuterVolumeSpecName: "console-config") pod "5009db96-8df4-4785-858a-0c2ab892c94a" (UID: "5009db96-8df4-4785-858a-0c2ab892c94a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:53:39.165827 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.165798 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-service-ca" (OuterVolumeSpecName: "service-ca") pod "5009db96-8df4-4785-858a-0c2ab892c94a" (UID: "5009db96-8df4-4785-858a-0c2ab892c94a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:53:39.165894 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.165820 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5009db96-8df4-4785-858a-0c2ab892c94a" (UID: "5009db96-8df4-4785-858a-0c2ab892c94a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:53:39.165894 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.165876 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-service-ca\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:53:39.166187 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.165893 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-console-config\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:53:39.166187 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.165904 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5009db96-8df4-4785-858a-0c2ab892c94a" (UID: "5009db96-8df4-4785-858a-0c2ab892c94a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:53:39.167348 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.167321 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5009db96-8df4-4785-858a-0c2ab892c94a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5009db96-8df4-4785-858a-0c2ab892c94a" (UID: "5009db96-8df4-4785-858a-0c2ab892c94a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:53:39.167453 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.167384 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5009db96-8df4-4785-858a-0c2ab892c94a-kube-api-access-sjz8w" (OuterVolumeSpecName: "kube-api-access-sjz8w") pod "5009db96-8df4-4785-858a-0c2ab892c94a" (UID: "5009db96-8df4-4785-858a-0c2ab892c94a"). InnerVolumeSpecName "kube-api-access-sjz8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:53:39.167453 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.167411 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5009db96-8df4-4785-858a-0c2ab892c94a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5009db96-8df4-4785-858a-0c2ab892c94a" (UID: "5009db96-8df4-4785-858a-0c2ab892c94a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:53:39.267354 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.267322 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5009db96-8df4-4785-858a-0c2ab892c94a-console-oauth-config\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:53:39.267354 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.267351 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjz8w\" (UniqueName: \"kubernetes.io/projected/5009db96-8df4-4785-858a-0c2ab892c94a-kube-api-access-sjz8w\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:53:39.267354 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.267362 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5009db96-8df4-4785-858a-0c2ab892c94a-console-serving-cert\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:53:39.267592 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.267372 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-oauth-serving-cert\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:53:39.267592 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.267380 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5009db96-8df4-4785-858a-0c2ab892c94a-trusted-ca-bundle\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:53:39.954825 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.954724 2562 generic.go:356] "Generic (PLEG): container finished" podID="5009db96-8df4-4785-858a-0c2ab892c94a" containerID="80adcf83691aad1229f47ed364721cf9fd92c059acf443a21057aed7e1a872be" exitCode=0 Jul 29 20:53:39.954825 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.954797 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bc6c7f784-bhnlb" Jul 29 20:53:39.954825 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.954811 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bc6c7f784-bhnlb" event={"ID":"5009db96-8df4-4785-858a-0c2ab892c94a","Type":"ContainerDied","Data":"80adcf83691aad1229f47ed364721cf9fd92c059acf443a21057aed7e1a872be"} Jul 29 20:53:39.955568 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.954899 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bc6c7f784-bhnlb" event={"ID":"5009db96-8df4-4785-858a-0c2ab892c94a","Type":"ContainerDied","Data":"c925fa1e0f720b21e095eeedea0dbf259215db9f1d67df529637f0e60478a1c8"} Jul 29 20:53:39.955568 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.954940 2562 scope.go:117] "RemoveContainer" containerID="80adcf83691aad1229f47ed364721cf9fd92c059acf443a21057aed7e1a872be" Jul 29 20:53:39.964596 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.964567 2562 scope.go:117] "RemoveContainer" containerID="80adcf83691aad1229f47ed364721cf9fd92c059acf443a21057aed7e1a872be" Jul 29 20:53:39.964869 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:39.964850 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80adcf83691aad1229f47ed364721cf9fd92c059acf443a21057aed7e1a872be\": container with ID starting with 80adcf83691aad1229f47ed364721cf9fd92c059acf443a21057aed7e1a872be not found: ID does not exist" containerID="80adcf83691aad1229f47ed364721cf9fd92c059acf443a21057aed7e1a872be" Jul 29 20:53:39.964933 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.964879 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80adcf83691aad1229f47ed364721cf9fd92c059acf443a21057aed7e1a872be"} err="failed to get container status \"80adcf83691aad1229f47ed364721cf9fd92c059acf443a21057aed7e1a872be\": rpc error: code = NotFound desc = could not find container \"80adcf83691aad1229f47ed364721cf9fd92c059acf443a21057aed7e1a872be\": container with ID starting with 80adcf83691aad1229f47ed364721cf9fd92c059acf443a21057aed7e1a872be not found: ID does not exist" Jul 29 20:53:39.971017 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.970986 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bc6c7f784-bhnlb"] Jul 29 20:53:39.974452 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:39.974426 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bc6c7f784-bhnlb"] Jul 29 20:53:41.557584 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:41.557549 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5009db96-8df4-4785-858a-0c2ab892c94a" path="/var/lib/kubelet/pods/5009db96-8df4-4785-858a-0c2ab892c94a/volumes" Jul 29 20:53:44.926787 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:44.926755 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4" Jul 29 20:53:44.927219 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:44.926832 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch" Jul 29 20:53:44.991451 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:44.991413 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4"] Jul 29 20:53:44.991668 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:44.991642 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4" podUID="acea6479-8752-4949-ac64-1d40a96e87ab" containerName="manager" containerID="cri-o://5ef2b0d8152761cd2f02b041d4a10f5396461b25ad69eec5de827e18dd602c00" gracePeriod=10 Jul 29 20:53:45.249187 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:45.249161 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4" Jul 29 20:53:45.324264 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:45.324227 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/acea6479-8752-4949-ac64-1d40a96e87ab-extensions-socket-volume\") pod \"acea6479-8752-4949-ac64-1d40a96e87ab\" (UID: \"acea6479-8752-4949-ac64-1d40a96e87ab\") " Jul 29 20:53:45.324435 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:45.324378 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9mfn\" (UniqueName: \"kubernetes.io/projected/acea6479-8752-4949-ac64-1d40a96e87ab-kube-api-access-s9mfn\") pod \"acea6479-8752-4949-ac64-1d40a96e87ab\" (UID: \"acea6479-8752-4949-ac64-1d40a96e87ab\") " Jul 29 20:53:45.324619 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:45.324595 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acea6479-8752-4949-ac64-1d40a96e87ab-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "acea6479-8752-4949-ac64-1d40a96e87ab" (UID: "acea6479-8752-4949-ac64-1d40a96e87ab"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 20:53:45.326446 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:45.326424 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acea6479-8752-4949-ac64-1d40a96e87ab-kube-api-access-s9mfn" (OuterVolumeSpecName: "kube-api-access-s9mfn") pod "acea6479-8752-4949-ac64-1d40a96e87ab" (UID: "acea6479-8752-4949-ac64-1d40a96e87ab"). InnerVolumeSpecName "kube-api-access-s9mfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:53:45.425220 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:45.425148 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9mfn\" (UniqueName: \"kubernetes.io/projected/acea6479-8752-4949-ac64-1d40a96e87ab-kube-api-access-s9mfn\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:53:45.425220 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:45.425177 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/acea6479-8752-4949-ac64-1d40a96e87ab-extensions-socket-volume\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:53:45.993701 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:45.993605 2562 generic.go:356] "Generic (PLEG): container finished" podID="acea6479-8752-4949-ac64-1d40a96e87ab" containerID="5ef2b0d8152761cd2f02b041d4a10f5396461b25ad69eec5de827e18dd602c00" exitCode=0 Jul 29 20:53:45.993701 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:45.993676 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4" Jul 29 20:53:45.994161 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:45.993684 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4" event={"ID":"acea6479-8752-4949-ac64-1d40a96e87ab","Type":"ContainerDied","Data":"5ef2b0d8152761cd2f02b041d4a10f5396461b25ad69eec5de827e18dd602c00"} Jul 29 20:53:45.994161 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:45.993783 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4" event={"ID":"acea6479-8752-4949-ac64-1d40a96e87ab","Type":"ContainerDied","Data":"725f6ca86d43133384fcbff04faa789909fec64d36c56320695f6c51d5bf6225"} Jul 29 20:53:45.994161 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:45.993804 2562 scope.go:117] "RemoveContainer" containerID="5ef2b0d8152761cd2f02b041d4a10f5396461b25ad69eec5de827e18dd602c00" Jul 29 20:53:46.003276 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:46.003258 2562 scope.go:117] "RemoveContainer" containerID="5ef2b0d8152761cd2f02b041d4a10f5396461b25ad69eec5de827e18dd602c00" Jul 29 20:53:46.003546 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:53:46.003524 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef2b0d8152761cd2f02b041d4a10f5396461b25ad69eec5de827e18dd602c00\": container with ID starting with 5ef2b0d8152761cd2f02b041d4a10f5396461b25ad69eec5de827e18dd602c00 not found: ID does not exist" containerID="5ef2b0d8152761cd2f02b041d4a10f5396461b25ad69eec5de827e18dd602c00" Jul 29 20:53:46.003623 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:46.003559 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef2b0d8152761cd2f02b041d4a10f5396461b25ad69eec5de827e18dd602c00"} err="failed to get container status \"5ef2b0d8152761cd2f02b041d4a10f5396461b25ad69eec5de827e18dd602c00\": rpc error: code = NotFound desc = could not find container \"5ef2b0d8152761cd2f02b041d4a10f5396461b25ad69eec5de827e18dd602c00\": container with ID starting with 5ef2b0d8152761cd2f02b041d4a10f5396461b25ad69eec5de827e18dd602c00 not found: ID does not exist" Jul 29 20:53:46.021976 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:46.021941 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4"] Jul 29 20:53:46.026946 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:46.026905 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqsm4"] Jul 29 20:53:47.557096 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:53:47.557060 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acea6479-8752-4949-ac64-1d40a96e87ab" path="/var/lib/kubelet/pods/acea6479-8752-4949-ac64-1d40a96e87ab/volumes" Jul 29 20:54:01.229545 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.229508 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g"] Jul 29 20:54:01.230094 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.229898 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acea6479-8752-4949-ac64-1d40a96e87ab" containerName="manager" Jul 29 20:54:01.230094 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.229909 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="acea6479-8752-4949-ac64-1d40a96e87ab" containerName="manager" Jul 29 20:54:01.230094 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.229940 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5009db96-8df4-4785-858a-0c2ab892c94a" containerName="console" Jul 29 20:54:01.230094 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.229946 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5009db96-8df4-4785-858a-0c2ab892c94a" containerName="console" Jul 29 20:54:01.230094 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.230012 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="acea6479-8752-4949-ac64-1d40a96e87ab" containerName="manager" Jul 29 20:54:01.230094 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.230025 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5009db96-8df4-4785-858a-0c2ab892c94a" containerName="console" Jul 29 20:54:01.234477 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.234455 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.245418 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.245387 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bfxr6\"" Jul 29 20:54:01.248190 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.248162 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g"] Jul 29 20:54:01.374961 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.374891 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.375149 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.375002 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.375149 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.375034 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.375149 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.375108 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.375263 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.375151 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.375263 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.375194 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.375263 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.375224 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.375263 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.375246 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bkgb\" (UniqueName: \"kubernetes.io/projected/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-kube-api-access-7bkgb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.375392 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.375263 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.476490 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.476446 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.476695 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.476501 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.476695 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.476545 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.476695 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.476582 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.476695 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.476612 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bkgb\" (UniqueName: \"kubernetes.io/projected/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-kube-api-access-7bkgb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.476695 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.476637 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.476695 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.476682 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.477084 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.476719 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.477084 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.476745 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.477084 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.476935 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.477084 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.477008 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.477264 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.477124 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.477264 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.477183 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.477347 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.477269 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.478975 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.478945 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.479145 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.479129 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.483693 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.483635 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.483800 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.483781 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bkgb\" (UniqueName: \"kubernetes.io/projected/fa5cd0c8-a7b7-45e0-9015-de4ad64b8219-kube-api-access-7bkgb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2229g\" (UID: \"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.547423 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.547387 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:01.702094 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:01.702038 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g"] Jul 29 20:54:02.092341 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:02.092301 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" event={"ID":"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219","Type":"ContainerStarted","Data":"2cf97c5e2135cd72557dee6b7dbaa6e1bb62e86d699e8ed2ecbedfaf4951f013"} Jul 29 20:54:05.197547 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:05.197509 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 20:54:05.197850 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:05.197580 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 20:54:05.197850 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:05.197611 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 20:54:06.117933 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:06.117888 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" event={"ID":"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219","Type":"ContainerStarted","Data":"16124d4257d1124e7e87e4200e887f978bf686c32daa643c6c4977a130052acb"} Jul 29 20:54:06.145082 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:06.145017 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" podStartSLOduration=1.660739621 podStartE2EDuration="5.144997482s" totalImagesPullingTime="3.484257861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:54:06.135232756 +0000 UTC m=+463.006336227" watchObservedRunningTime="2026-07-29 20:54:06.144997482 +0000 UTC m=+463.016100949" Jul 29 20:54:06.548271 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:06.548171 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:06.553659 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:06.553628 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:07.072200 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:07.072154 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-xpg58"] Jul 29 20:54:07.076084 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:07.076058 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-xpg58" Jul 29 20:54:07.077948 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:07.077907 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 29 20:54:07.078089 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:07.078022 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wdbm6\"" Jul 29 20:54:07.082838 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:07.082813 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-xpg58"] Jul 29 20:54:07.124335 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:07.124296 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:07.125326 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:07.125303 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:54:07.128699 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:07.128669 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ac35b0c8-1f93-4150-a8cf-abedd321149f-config-file\") pod \"limitador-limitador-64c94fd64f-xpg58\" (UID: \"ac35b0c8-1f93-4150-a8cf-abedd321149f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-xpg58" Jul 29 20:54:07.128837 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:07.128716 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmcqf\" (UniqueName: \"kubernetes.io/projected/ac35b0c8-1f93-4150-a8cf-abedd321149f-kube-api-access-lmcqf\") pod \"limitador-limitador-64c94fd64f-xpg58\" (UID: \"ac35b0c8-1f93-4150-a8cf-abedd321149f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-xpg58" Jul 29 20:54:07.168871 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:07.168797 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-xpg58"] Jul 29 20:54:07.229996 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:07.229948 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ac35b0c8-1f93-4150-a8cf-abedd321149f-config-file\") pod \"limitador-limitador-64c94fd64f-xpg58\" (UID: \"ac35b0c8-1f93-4150-a8cf-abedd321149f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-xpg58" Jul 29 20:54:07.229996 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:07.229986 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmcqf\" (UniqueName: \"kubernetes.io/projected/ac35b0c8-1f93-4150-a8cf-abedd321149f-kube-api-access-lmcqf\") pod \"limitador-limitador-64c94fd64f-xpg58\" (UID: \"ac35b0c8-1f93-4150-a8cf-abedd321149f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-xpg58" Jul 29 20:54:07.230663 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:07.230639 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ac35b0c8-1f93-4150-a8cf-abedd321149f-config-file\") pod \"limitador-limitador-64c94fd64f-xpg58\" (UID: \"ac35b0c8-1f93-4150-a8cf-abedd321149f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-xpg58" Jul 29 20:54:07.237323 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:07.237299 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmcqf\" (UniqueName: \"kubernetes.io/projected/ac35b0c8-1f93-4150-a8cf-abedd321149f-kube-api-access-lmcqf\") pod \"limitador-limitador-64c94fd64f-xpg58\" (UID: \"ac35b0c8-1f93-4150-a8cf-abedd321149f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-xpg58" Jul 29 20:54:07.388545 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:07.388439 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-xpg58" Jul 29 20:54:07.535788 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:07.535760 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-xpg58"] Jul 29 20:54:07.536825 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:54:07.536795 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac35b0c8_1f93_4150_a8cf_abedd321149f.slice/crio-fe0d9fd887f9814325513bbe34615b33e14d3708dbeb37ac7fb1da22bf6bdd69 WatchSource:0}: Error finding container fe0d9fd887f9814325513bbe34615b33e14d3708dbeb37ac7fb1da22bf6bdd69: Status 404 returned error can't find the container with id fe0d9fd887f9814325513bbe34615b33e14d3708dbeb37ac7fb1da22bf6bdd69 Jul 29 20:54:08.134429 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:08.134381 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-xpg58" event={"ID":"ac35b0c8-1f93-4150-a8cf-abedd321149f","Type":"ContainerStarted","Data":"fe0d9fd887f9814325513bbe34615b33e14d3708dbeb37ac7fb1da22bf6bdd69"} Jul 29 20:54:11.154399 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:11.154360 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-xpg58" event={"ID":"ac35b0c8-1f93-4150-a8cf-abedd321149f","Type":"ContainerStarted","Data":"d4de23e1169eff9cb6798c9c29ad19df9b4d75431fe6a1896960364cd44b3eaa"} Jul 29 20:54:11.154868 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:11.154456 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-xpg58" Jul 29 20:54:11.170735 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:11.170538 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-xpg58" podStartSLOduration=1.554775356 podStartE2EDuration="4.17051816s" totalImagesPullingTime="2.615742804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:54:11.170167224 +0000 UTC m=+468.041270694" watchObservedRunningTime="2026-07-29 20:54:11.17051816 +0000 UTC m=+468.041621628" Jul 29 20:54:22.161944 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:22.161883 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-xpg58" Jul 29 20:54:23.526116 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:23.526084 2562 scope.go:117] "RemoveContainer" containerID="0b737f2f659b6aa832474270e109c6dcf82980e5d43cd9c0f5dfab63e5d3db6b" Jul 29 20:54:23.535006 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:23.534982 2562 scope.go:117] "RemoveContainer" containerID="84e7f355a92d174b8b898653580e9551bc8f788e52f6e70c234f70d814fd374d" Jul 29 20:54:23.543224 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:23.543199 2562 scope.go:117] "RemoveContainer" containerID="a00cb10010c66e4afc640e30b06abff6d2c8986c049fdda13fcacf6f48540480" Jul 29 20:54:23.552378 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:23.552354 2562 scope.go:117] "RemoveContainer" containerID="cc79d8eb057249931dc8707c73aa3faad29d2d14b1e46301ef658c09bef1072a" Jul 29 20:54:23.561885 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:23.561863 2562 scope.go:117] "RemoveContainer" containerID="3b40e92336cf7b8b5104dec082302f6b08466ead39b6b9998e9249dbe0312ff9" Jul 29 20:54:23.570130 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:23.570105 2562 scope.go:117] "RemoveContainer" containerID="8327c154ba2f4110332b0a54274de9aa3ea4c64fbc11bd07b00a4ebed5b58941" Jul 29 20:54:23.578701 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:23.578681 2562 scope.go:117] "RemoveContainer" containerID="b5cd54c95996e2f0b1290fe446ddaf0349ca22dece2cc86dd1aaf966ebd5c29b" Jul 29 20:54:23.939782 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:23.939741 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-xpg58"] Jul 29 20:54:23.940092 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:23.940050 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-xpg58" podUID="ac35b0c8-1f93-4150-a8cf-abedd321149f" containerName="limitador" containerID="cri-o://d4de23e1169eff9cb6798c9c29ad19df9b4d75431fe6a1896960364cd44b3eaa" gracePeriod=30 Jul 29 20:54:24.497279 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:24.497245 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-xpg58" Jul 29 20:54:24.595976 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:24.595882 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ac35b0c8-1f93-4150-a8cf-abedd321149f-config-file\") pod \"ac35b0c8-1f93-4150-a8cf-abedd321149f\" (UID: \"ac35b0c8-1f93-4150-a8cf-abedd321149f\") " Jul 29 20:54:24.595976 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:24.595977 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmcqf\" (UniqueName: \"kubernetes.io/projected/ac35b0c8-1f93-4150-a8cf-abedd321149f-kube-api-access-lmcqf\") pod \"ac35b0c8-1f93-4150-a8cf-abedd321149f\" (UID: \"ac35b0c8-1f93-4150-a8cf-abedd321149f\") " Jul 29 20:54:24.596425 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:24.596261 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac35b0c8-1f93-4150-a8cf-abedd321149f-config-file" (OuterVolumeSpecName: "config-file") pod "ac35b0c8-1f93-4150-a8cf-abedd321149f" (UID: "ac35b0c8-1f93-4150-a8cf-abedd321149f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 20:54:24.598060 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:24.598034 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac35b0c8-1f93-4150-a8cf-abedd321149f-kube-api-access-lmcqf" (OuterVolumeSpecName: "kube-api-access-lmcqf") pod "ac35b0c8-1f93-4150-a8cf-abedd321149f" (UID: "ac35b0c8-1f93-4150-a8cf-abedd321149f"). InnerVolumeSpecName "kube-api-access-lmcqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:54:24.696891 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:24.696856 2562 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ac35b0c8-1f93-4150-a8cf-abedd321149f-config-file\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:54:24.696891 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:24.696890 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmcqf\" (UniqueName: \"kubernetes.io/projected/ac35b0c8-1f93-4150-a8cf-abedd321149f-kube-api-access-lmcqf\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:54:25.238125 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:25.238087 2562 generic.go:356] "Generic (PLEG): container finished" podID="ac35b0c8-1f93-4150-a8cf-abedd321149f" containerID="d4de23e1169eff9cb6798c9c29ad19df9b4d75431fe6a1896960364cd44b3eaa" exitCode=0 Jul 29 20:54:25.238125 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:25.238136 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-xpg58" Jul 29 20:54:25.238353 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:25.238174 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-xpg58" event={"ID":"ac35b0c8-1f93-4150-a8cf-abedd321149f","Type":"ContainerDied","Data":"d4de23e1169eff9cb6798c9c29ad19df9b4d75431fe6a1896960364cd44b3eaa"} Jul 29 20:54:25.238353 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:25.238232 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-xpg58" event={"ID":"ac35b0c8-1f93-4150-a8cf-abedd321149f","Type":"ContainerDied","Data":"fe0d9fd887f9814325513bbe34615b33e14d3708dbeb37ac7fb1da22bf6bdd69"} Jul 29 20:54:25.238353 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:25.238256 2562 scope.go:117] "RemoveContainer" containerID="d4de23e1169eff9cb6798c9c29ad19df9b4d75431fe6a1896960364cd44b3eaa" Jul 29 20:54:25.248139 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:25.248119 2562 scope.go:117] "RemoveContainer" containerID="d4de23e1169eff9cb6798c9c29ad19df9b4d75431fe6a1896960364cd44b3eaa" Jul 29 20:54:25.248434 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:54:25.248416 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4de23e1169eff9cb6798c9c29ad19df9b4d75431fe6a1896960364cd44b3eaa\": container with ID starting with d4de23e1169eff9cb6798c9c29ad19df9b4d75431fe6a1896960364cd44b3eaa not found: ID does not exist" containerID="d4de23e1169eff9cb6798c9c29ad19df9b4d75431fe6a1896960364cd44b3eaa" Jul 29 20:54:25.248488 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:25.248442 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4de23e1169eff9cb6798c9c29ad19df9b4d75431fe6a1896960364cd44b3eaa"} err="failed to get container status \"d4de23e1169eff9cb6798c9c29ad19df9b4d75431fe6a1896960364cd44b3eaa\": rpc error: code = NotFound desc = could not find container \"d4de23e1169eff9cb6798c9c29ad19df9b4d75431fe6a1896960364cd44b3eaa\": container with ID starting with d4de23e1169eff9cb6798c9c29ad19df9b4d75431fe6a1896960364cd44b3eaa not found: ID does not exist" Jul 29 20:54:25.258812 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:25.258781 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-xpg58"] Jul 29 20:54:25.261102 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:25.261077 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-xpg58"] Jul 29 20:54:25.557595 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:25.557561 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac35b0c8-1f93-4150-a8cf-abedd321149f" path="/var/lib/kubelet/pods/ac35b0c8-1f93-4150-a8cf-abedd321149f/volumes" Jul 29 20:54:28.656838 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:28.656805 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-db5sk"] Jul 29 20:54:28.657251 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:28.657241 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac35b0c8-1f93-4150-a8cf-abedd321149f" containerName="limitador" Jul 29 20:54:28.657292 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:28.657255 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac35b0c8-1f93-4150-a8cf-abedd321149f" containerName="limitador" Jul 29 20:54:28.657332 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:28.657320 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac35b0c8-1f93-4150-a8cf-abedd321149f" containerName="limitador" Jul 29 20:54:28.661973 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:28.661946 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-db5sk" Jul 29 20:54:28.664899 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:28.664875 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jx6md\"" Jul 29 20:54:28.665086 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:28.664912 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 20:54:28.665269 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:28.665243 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 29 20:54:28.665415 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:28.665301 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 29 20:54:28.675636 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:28.675608 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-db5sk"] Jul 29 20:54:28.735420 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:28.735377 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c833efa9-365d-4d5b-b4c1-2a114f077fb2-data\") pod \"postgres-69fb4594cf-db5sk\" (UID: \"c833efa9-365d-4d5b-b4c1-2a114f077fb2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-db5sk" Jul 29 20:54:28.735420 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:28.735426 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58vq\" (UniqueName: \"kubernetes.io/projected/c833efa9-365d-4d5b-b4c1-2a114f077fb2-kube-api-access-d58vq\") pod \"postgres-69fb4594cf-db5sk\" (UID: \"c833efa9-365d-4d5b-b4c1-2a114f077fb2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-db5sk" Jul 29 20:54:28.836548 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:28.836502 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c833efa9-365d-4d5b-b4c1-2a114f077fb2-data\") pod \"postgres-69fb4594cf-db5sk\" (UID: \"c833efa9-365d-4d5b-b4c1-2a114f077fb2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-db5sk" Jul 29 20:54:28.836735 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:28.836560 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58vq\" (UniqueName: \"kubernetes.io/projected/c833efa9-365d-4d5b-b4c1-2a114f077fb2-kube-api-access-d58vq\") pod \"postgres-69fb4594cf-db5sk\" (UID: \"c833efa9-365d-4d5b-b4c1-2a114f077fb2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-db5sk" Jul 29 20:54:28.837036 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:28.837012 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c833efa9-365d-4d5b-b4c1-2a114f077fb2-data\") pod \"postgres-69fb4594cf-db5sk\" (UID: \"c833efa9-365d-4d5b-b4c1-2a114f077fb2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-db5sk" Jul 29 20:54:28.844242 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:28.844215 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58vq\" (UniqueName: \"kubernetes.io/projected/c833efa9-365d-4d5b-b4c1-2a114f077fb2-kube-api-access-d58vq\") pod \"postgres-69fb4594cf-db5sk\" (UID: \"c833efa9-365d-4d5b-b4c1-2a114f077fb2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-db5sk" Jul 29 20:54:28.974108 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:28.974023 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-db5sk" Jul 29 20:54:29.111230 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:29.111196 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-db5sk"] Jul 29 20:54:29.112646 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:54:29.112618 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc833efa9_365d_4d5b_b4c1_2a114f077fb2.slice/crio-8a660c37b7098d4a1efe1cdcc3a7073921f4d57ab40871e5b2e7667e0f2e6601 WatchSource:0}: Error finding container 8a660c37b7098d4a1efe1cdcc3a7073921f4d57ab40871e5b2e7667e0f2e6601: Status 404 returned error can't find the container with id 8a660c37b7098d4a1efe1cdcc3a7073921f4d57ab40871e5b2e7667e0f2e6601 Jul 29 20:54:29.261148 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:29.261056 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-db5sk" event={"ID":"c833efa9-365d-4d5b-b4c1-2a114f077fb2","Type":"ContainerStarted","Data":"8a660c37b7098d4a1efe1cdcc3a7073921f4d57ab40871e5b2e7667e0f2e6601"} Jul 29 20:54:34.297073 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:34.296963 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-db5sk" event={"ID":"c833efa9-365d-4d5b-b4c1-2a114f077fb2","Type":"ContainerStarted","Data":"74c507ebef6c43ddb38b17efff7f60315ceca02a6529b6d001a22d23eb74061c"} Jul 29 20:54:34.297521 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:34.297219 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-db5sk" Jul 29 20:54:34.313448 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:34.313387 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-db5sk" podStartSLOduration=1.488567054 podStartE2EDuration="6.313373154s" totalImagesPullingTime="4.8248061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:54:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:54:34.310750779 +0000 UTC m=+491.181854247" watchObservedRunningTime="2026-07-29 20:54:34.313373154 +0000 UTC m=+491.184476620" Jul 29 20:54:40.332381 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:40.332351 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-db5sk" Jul 29 20:54:47.390645 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.390609 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-856ddc6576-8qpv4"] Jul 29 20:54:47.394194 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.394174 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-856ddc6576-8qpv4" Jul 29 20:54:47.396356 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.396328 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 20:54:47.396802 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.396777 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 20:54:47.396932 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.396892 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pvk7c\"" Jul 29 20:54:47.404121 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.404098 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/425a10a1-7e60-416b-927e-5e1de7423dbd-webhook-cert\") pod \"maas-controller-856ddc6576-8qpv4\" (UID: \"425a10a1-7e60-416b-927e-5e1de7423dbd\") " pod="opendatahub/maas-controller-856ddc6576-8qpv4" Jul 29 20:54:47.404221 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.404129 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9kjm\" (UniqueName: \"kubernetes.io/projected/425a10a1-7e60-416b-927e-5e1de7423dbd-kube-api-access-h9kjm\") pod \"maas-controller-856ddc6576-8qpv4\" (UID: \"425a10a1-7e60-416b-927e-5e1de7423dbd\") " pod="opendatahub/maas-controller-856ddc6576-8qpv4" Jul 29 20:54:47.409227 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.409201 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-856ddc6576-8qpv4"] Jul 29 20:54:47.505097 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.505055 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9kjm\" (UniqueName: \"kubernetes.io/projected/425a10a1-7e60-416b-927e-5e1de7423dbd-kube-api-access-h9kjm\") pod \"maas-controller-856ddc6576-8qpv4\" (UID: \"425a10a1-7e60-416b-927e-5e1de7423dbd\") " pod="opendatahub/maas-controller-856ddc6576-8qpv4" Jul 29 20:54:47.505293 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.505173 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/425a10a1-7e60-416b-927e-5e1de7423dbd-webhook-cert\") pod \"maas-controller-856ddc6576-8qpv4\" (UID: \"425a10a1-7e60-416b-927e-5e1de7423dbd\") " pod="opendatahub/maas-controller-856ddc6576-8qpv4" Jul 29 20:54:47.507621 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.507592 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/425a10a1-7e60-416b-927e-5e1de7423dbd-webhook-cert\") pod \"maas-controller-856ddc6576-8qpv4\" (UID: \"425a10a1-7e60-416b-927e-5e1de7423dbd\") " pod="opendatahub/maas-controller-856ddc6576-8qpv4" Jul 29 20:54:47.520872 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.520837 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9kjm\" (UniqueName: \"kubernetes.io/projected/425a10a1-7e60-416b-927e-5e1de7423dbd-kube-api-access-h9kjm\") pod \"maas-controller-856ddc6576-8qpv4\" (UID: \"425a10a1-7e60-416b-927e-5e1de7423dbd\") " pod="opendatahub/maas-controller-856ddc6576-8qpv4" Jul 29 20:54:47.608404 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.608372 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c85847677-qgrnz"] Jul 29 20:54:47.612020 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.611997 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c85847677-qgrnz" Jul 29 20:54:47.627080 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.627049 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c85847677-qgrnz"] Jul 29 20:54:47.704652 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.704551 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-856ddc6576-8qpv4" Jul 29 20:54:47.707073 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.707044 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f605d20b-2271-47c8-b61e-9a5a70e51121-webhook-cert\") pod \"maas-controller-5c85847677-qgrnz\" (UID: \"f605d20b-2271-47c8-b61e-9a5a70e51121\") " pod="opendatahub/maas-controller-5c85847677-qgrnz" Jul 29 20:54:47.707191 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.707092 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr7qd\" (UniqueName: \"kubernetes.io/projected/f605d20b-2271-47c8-b61e-9a5a70e51121-kube-api-access-qr7qd\") pod \"maas-controller-5c85847677-qgrnz\" (UID: \"f605d20b-2271-47c8-b61e-9a5a70e51121\") " pod="opendatahub/maas-controller-5c85847677-qgrnz" Jul 29 20:54:47.807604 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.807564 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f605d20b-2271-47c8-b61e-9a5a70e51121-webhook-cert\") pod \"maas-controller-5c85847677-qgrnz\" (UID: \"f605d20b-2271-47c8-b61e-9a5a70e51121\") " pod="opendatahub/maas-controller-5c85847677-qgrnz" Jul 29 20:54:47.807772 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.807631 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr7qd\" (UniqueName: \"kubernetes.io/projected/f605d20b-2271-47c8-b61e-9a5a70e51121-kube-api-access-qr7qd\") pod \"maas-controller-5c85847677-qgrnz\" (UID: \"f605d20b-2271-47c8-b61e-9a5a70e51121\") " pod="opendatahub/maas-controller-5c85847677-qgrnz" Jul 29 20:54:47.810395 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.810358 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f605d20b-2271-47c8-b61e-9a5a70e51121-webhook-cert\") pod \"maas-controller-5c85847677-qgrnz\" (UID: \"f605d20b-2271-47c8-b61e-9a5a70e51121\") " pod="opendatahub/maas-controller-5c85847677-qgrnz" Jul 29 20:54:47.817068 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.816264 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr7qd\" (UniqueName: \"kubernetes.io/projected/f605d20b-2271-47c8-b61e-9a5a70e51121-kube-api-access-qr7qd\") pod \"maas-controller-5c85847677-qgrnz\" (UID: \"f605d20b-2271-47c8-b61e-9a5a70e51121\") " pod="opendatahub/maas-controller-5c85847677-qgrnz" Jul 29 20:54:47.847323 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.847288 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-856ddc6576-8qpv4"] Jul 29 20:54:47.848547 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:54:47.848522 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425a10a1_7e60_416b_927e_5e1de7423dbd.slice/crio-f7c3f6a145ab3c3edcec1f9628bcd6f6a26c3f78549946370d5a50ca2c52fa63 WatchSource:0}: Error finding container f7c3f6a145ab3c3edcec1f9628bcd6f6a26c3f78549946370d5a50ca2c52fa63: Status 404 returned error can't find the container with id f7c3f6a145ab3c3edcec1f9628bcd6f6a26c3f78549946370d5a50ca2c52fa63 Jul 29 20:54:47.922770 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:47.922731 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c85847677-qgrnz" Jul 29 20:54:48.061323 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:48.061296 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c85847677-qgrnz"] Jul 29 20:54:48.062668 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:54:48.062633 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf605d20b_2271_47c8_b61e_9a5a70e51121.slice/crio-01d9cc4a10e0982b5502cef363240a7d42e69921a0eac39e3bc6cb7b2535c7a6 WatchSource:0}: Error finding container 01d9cc4a10e0982b5502cef363240a7d42e69921a0eac39e3bc6cb7b2535c7a6: Status 404 returned error can't find the container with id 01d9cc4a10e0982b5502cef363240a7d42e69921a0eac39e3bc6cb7b2535c7a6 Jul 29 20:54:48.380083 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:48.380041 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-856ddc6576-8qpv4" event={"ID":"425a10a1-7e60-416b-927e-5e1de7423dbd","Type":"ContainerStarted","Data":"f7c3f6a145ab3c3edcec1f9628bcd6f6a26c3f78549946370d5a50ca2c52fa63"} Jul 29 20:54:48.381297 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:48.381270 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c85847677-qgrnz" event={"ID":"f605d20b-2271-47c8-b61e-9a5a70e51121","Type":"ContainerStarted","Data":"01d9cc4a10e0982b5502cef363240a7d42e69921a0eac39e3bc6cb7b2535c7a6"} Jul 29 20:54:51.403003 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:51.402957 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-856ddc6576-8qpv4" event={"ID":"425a10a1-7e60-416b-927e-5e1de7423dbd","Type":"ContainerStarted","Data":"ee27ed449a50e09506f89b87bbeaa45129433b10037fbf1c14307915067b7110"} Jul 29 20:54:51.403500 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:51.403043 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-856ddc6576-8qpv4" Jul 29 20:54:51.404611 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:51.404589 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c85847677-qgrnz" event={"ID":"f605d20b-2271-47c8-b61e-9a5a70e51121","Type":"ContainerStarted","Data":"b4766c482ffd12e0769c8c9a4dcaceacfb724f2a1651b66b4433c75815edc63f"} Jul 29 20:54:51.404701 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:51.404657 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c85847677-qgrnz" Jul 29 20:54:51.418136 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:51.418075 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-856ddc6576-8qpv4" podStartSLOduration=1.890980804 podStartE2EDuration="4.418055758s" totalImagesPullingTime="2.527074954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:54:51.417098847 +0000 UTC m=+508.288202315" watchObservedRunningTime="2026-07-29 20:54:51.418055758 +0000 UTC m=+508.289159226" Jul 29 20:54:51.434301 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:54:51.434244 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c85847677-qgrnz" podStartSLOduration=2.113262598 podStartE2EDuration="4.434227296s" totalImagesPullingTime="2.320964698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:54:51.431032902 +0000 UTC m=+508.302136371" watchObservedRunningTime="2026-07-29 20:54:51.434227296 +0000 UTC m=+508.305330812" Jul 29 20:55:02.413620 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:02.413588 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c85847677-qgrnz" Jul 29 20:55:02.414049 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:02.413661 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-856ddc6576-8qpv4" Jul 29 20:55:02.468487 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:02.468451 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-856ddc6576-8qpv4"] Jul 29 20:55:02.475180 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:02.475131 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-856ddc6576-8qpv4" podUID="425a10a1-7e60-416b-927e-5e1de7423dbd" containerName="manager" containerID="cri-o://ee27ed449a50e09506f89b87bbeaa45129433b10037fbf1c14307915067b7110" gracePeriod=30 Jul 29 20:55:02.732276 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:02.732242 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-856ddc6576-8qpv4" Jul 29 20:55:02.846077 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:02.846036 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/425a10a1-7e60-416b-927e-5e1de7423dbd-webhook-cert\") pod \"425a10a1-7e60-416b-927e-5e1de7423dbd\" (UID: \"425a10a1-7e60-416b-927e-5e1de7423dbd\") " Jul 29 20:55:02.846077 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:02.846081 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9kjm\" (UniqueName: \"kubernetes.io/projected/425a10a1-7e60-416b-927e-5e1de7423dbd-kube-api-access-h9kjm\") pod \"425a10a1-7e60-416b-927e-5e1de7423dbd\" (UID: \"425a10a1-7e60-416b-927e-5e1de7423dbd\") " Jul 29 20:55:02.848335 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:02.848302 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425a10a1-7e60-416b-927e-5e1de7423dbd-kube-api-access-h9kjm" (OuterVolumeSpecName: "kube-api-access-h9kjm") pod "425a10a1-7e60-416b-927e-5e1de7423dbd" (UID: "425a10a1-7e60-416b-927e-5e1de7423dbd"). InnerVolumeSpecName "kube-api-access-h9kjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:55:02.848469 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:02.848447 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/425a10a1-7e60-416b-927e-5e1de7423dbd-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "425a10a1-7e60-416b-927e-5e1de7423dbd" (UID: "425a10a1-7e60-416b-927e-5e1de7423dbd"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:55:02.947095 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:02.947046 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9kjm\" (UniqueName: \"kubernetes.io/projected/425a10a1-7e60-416b-927e-5e1de7423dbd-kube-api-access-h9kjm\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:55:02.947095 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:02.947084 2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/425a10a1-7e60-416b-927e-5e1de7423dbd-webhook-cert\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:55:03.482307 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:03.482267 2562 generic.go:356] "Generic (PLEG): container finished" podID="425a10a1-7e60-416b-927e-5e1de7423dbd" containerID="ee27ed449a50e09506f89b87bbeaa45129433b10037fbf1c14307915067b7110" exitCode=0 Jul 29 20:55:03.482733 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:03.482313 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-856ddc6576-8qpv4" Jul 29 20:55:03.482733 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:03.482355 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-856ddc6576-8qpv4" event={"ID":"425a10a1-7e60-416b-927e-5e1de7423dbd","Type":"ContainerDied","Data":"ee27ed449a50e09506f89b87bbeaa45129433b10037fbf1c14307915067b7110"} Jul 29 20:55:03.482733 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:03.482411 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-856ddc6576-8qpv4" event={"ID":"425a10a1-7e60-416b-927e-5e1de7423dbd","Type":"ContainerDied","Data":"f7c3f6a145ab3c3edcec1f9628bcd6f6a26c3f78549946370d5a50ca2c52fa63"} Jul 29 20:55:03.482733 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:03.482444 2562 scope.go:117] "RemoveContainer" containerID="ee27ed449a50e09506f89b87bbeaa45129433b10037fbf1c14307915067b7110" Jul 29 20:55:03.492659 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:03.492634 2562 scope.go:117] "RemoveContainer" containerID="ee27ed449a50e09506f89b87bbeaa45129433b10037fbf1c14307915067b7110" Jul 29 20:55:03.492997 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:55:03.492978 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee27ed449a50e09506f89b87bbeaa45129433b10037fbf1c14307915067b7110\": container with ID starting with ee27ed449a50e09506f89b87bbeaa45129433b10037fbf1c14307915067b7110 not found: ID does not exist" containerID="ee27ed449a50e09506f89b87bbeaa45129433b10037fbf1c14307915067b7110" Jul 29 20:55:03.493072 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:03.493010 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee27ed449a50e09506f89b87bbeaa45129433b10037fbf1c14307915067b7110"} err="failed to get container status \"ee27ed449a50e09506f89b87bbeaa45129433b10037fbf1c14307915067b7110\": rpc error: code = NotFound desc = could not find container \"ee27ed449a50e09506f89b87bbeaa45129433b10037fbf1c14307915067b7110\": container with ID starting with ee27ed449a50e09506f89b87bbeaa45129433b10037fbf1c14307915067b7110 not found: ID does not exist" Jul 29 20:55:03.503451 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:03.503409 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-856ddc6576-8qpv4"] Jul 29 20:55:03.507052 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:03.507018 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-856ddc6576-8qpv4"] Jul 29 20:55:03.557796 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:03.557759 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="425a10a1-7e60-416b-927e-5e1de7423dbd" path="/var/lib/kubelet/pods/425a10a1-7e60-416b-927e-5e1de7423dbd/volumes" Jul 29 20:55:23.951574 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:23.951490 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg"] Jul 29 20:55:23.952039 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:23.951987 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="425a10a1-7e60-416b-927e-5e1de7423dbd" containerName="manager" Jul 29 20:55:23.952039 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:23.952002 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="425a10a1-7e60-416b-927e-5e1de7423dbd" containerName="manager" Jul 29 20:55:23.952137 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:23.952091 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="425a10a1-7e60-416b-927e-5e1de7423dbd" containerName="manager" Jul 29 20:55:23.957373 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:23.957351 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg" Jul 29 20:55:23.959318 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:23.959289 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 29 20:55:23.959444 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:23.959289 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-944pn\"" Jul 29 20:55:23.964894 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:23.964866 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg"] Jul 29 20:55:24.058060 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.058023 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wnlj\" (UniqueName: \"kubernetes.io/projected/34716149-c1ff-4198-b465-149d7c0e8d14-kube-api-access-4wnlj\") pod \"maas-api-848d99f5fb-d85qg\" (UID: \"34716149-c1ff-4198-b465-149d7c0e8d14\") " pod="odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg" Jul 29 20:55:24.058261 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.058069 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34716149-c1ff-4198-b465-149d7c0e8d14-openshift-service-ca\") pod \"maas-api-848d99f5fb-d85qg\" (UID: \"34716149-c1ff-4198-b465-149d7c0e8d14\") " pod="odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg" Jul 29 20:55:24.058261 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.058213 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34716149-c1ff-4198-b465-149d7c0e8d14-maas-api-tls\") pod \"maas-api-848d99f5fb-d85qg\" (UID: \"34716149-c1ff-4198-b465-149d7c0e8d14\") " pod="odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg" Jul 29 20:55:24.159390 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.159347 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wnlj\" (UniqueName: \"kubernetes.io/projected/34716149-c1ff-4198-b465-149d7c0e8d14-kube-api-access-4wnlj\") pod \"maas-api-848d99f5fb-d85qg\" (UID: \"34716149-c1ff-4198-b465-149d7c0e8d14\") " pod="odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg" Jul 29 20:55:24.159390 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.159395 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34716149-c1ff-4198-b465-149d7c0e8d14-openshift-service-ca\") pod \"maas-api-848d99f5fb-d85qg\" (UID: \"34716149-c1ff-4198-b465-149d7c0e8d14\") " pod="odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg" Jul 29 20:55:24.159685 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.159464 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34716149-c1ff-4198-b465-149d7c0e8d14-maas-api-tls\") pod \"maas-api-848d99f5fb-d85qg\" (UID: \"34716149-c1ff-4198-b465-149d7c0e8d14\") " pod="odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg" Jul 29 20:55:24.159685 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:55:24.159575 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 29 20:55:24.159685 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:55:24.159644 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34716149-c1ff-4198-b465-149d7c0e8d14-maas-api-tls podName:34716149-c1ff-4198-b465-149d7c0e8d14 nodeName:}" failed. No retries permitted until 2026-07-29 20:55:24.659622188 +0000 UTC m=+541.530725635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34716149-c1ff-4198-b465-149d7c0e8d14-maas-api-tls") pod "maas-api-848d99f5fb-d85qg" (UID: "34716149-c1ff-4198-b465-149d7c0e8d14") : secret "maas-api-serving-cert" not found Jul 29 20:55:24.160244 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.160224 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34716149-c1ff-4198-b465-149d7c0e8d14-openshift-service-ca\") pod \"maas-api-848d99f5fb-d85qg\" (UID: \"34716149-c1ff-4198-b465-149d7c0e8d14\") " pod="odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg" Jul 29 20:55:24.170775 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.170743 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wnlj\" (UniqueName: \"kubernetes.io/projected/34716149-c1ff-4198-b465-149d7c0e8d14-kube-api-access-4wnlj\") pod \"maas-api-848d99f5fb-d85qg\" (UID: \"34716149-c1ff-4198-b465-149d7c0e8d14\") " pod="odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg" Jul 29 20:55:24.479691 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.479653 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-cdblw"] Jul 29 20:55:24.483706 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.483687 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cdblw" Jul 29 20:55:24.485679 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.485651 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 29 20:55:24.485801 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.485771 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-sssz9\"" Jul 29 20:55:24.497166 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.497137 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-cdblw"] Jul 29 20:55:24.501497 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.501461 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-mmtk9"] Jul 29 20:55:24.510658 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.509904 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-mmtk9" Jul 29 20:55:24.516669 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.516639 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-mmtk9"] Jul 29 20:55:24.564701 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.564652 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49zr6\" (UniqueName: \"kubernetes.io/projected/24fdea1b-1074-47cf-9b52-d39f228fa43b-kube-api-access-49zr6\") pod \"payload-processing-5cc6cd444b-mmtk9\" (UID: \"24fdea1b-1074-47cf-9b52-d39f228fa43b\") " pod="openshift-ingress/payload-processing-5cc6cd444b-mmtk9" Jul 29 20:55:24.564911 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.564754 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372a884b-b391-42b9-9299-cb2173cbabe1-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-cdblw\" (UID: \"372a884b-b391-42b9-9299-cb2173cbabe1\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cdblw" Jul 29 20:55:24.564911 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.564798 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24fdea1b-1074-47cf-9b52-d39f228fa43b-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-mmtk9\" (UID: \"24fdea1b-1074-47cf-9b52-d39f228fa43b\") " pod="openshift-ingress/payload-processing-5cc6cd444b-mmtk9" Jul 29 20:55:24.565035 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.564912 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bwkq\" (UniqueName: \"kubernetes.io/projected/372a884b-b391-42b9-9299-cb2173cbabe1-kube-api-access-7bwkq\") pod \"payload-pre-processing-687dbb6cf6-cdblw\" (UID: \"372a884b-b391-42b9-9299-cb2173cbabe1\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cdblw" Jul 29 20:55:24.665910 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.665874 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24fdea1b-1074-47cf-9b52-d39f228fa43b-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-mmtk9\" (UID: \"24fdea1b-1074-47cf-9b52-d39f228fa43b\") " pod="openshift-ingress/payload-processing-5cc6cd444b-mmtk9" Jul 29 20:55:24.666126 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.666026 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bwkq\" (UniqueName: \"kubernetes.io/projected/372a884b-b391-42b9-9299-cb2173cbabe1-kube-api-access-7bwkq\") pod \"payload-pre-processing-687dbb6cf6-cdblw\" (UID: \"372a884b-b391-42b9-9299-cb2173cbabe1\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cdblw" Jul 29 20:55:24.666126 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.666080 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34716149-c1ff-4198-b465-149d7c0e8d14-maas-api-tls\") pod \"maas-api-848d99f5fb-d85qg\" (UID: \"34716149-c1ff-4198-b465-149d7c0e8d14\") " pod="odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg" Jul 29 20:55:24.666126 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.666110 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49zr6\" (UniqueName: \"kubernetes.io/projected/24fdea1b-1074-47cf-9b52-d39f228fa43b-kube-api-access-49zr6\") pod \"payload-processing-5cc6cd444b-mmtk9\" (UID: \"24fdea1b-1074-47cf-9b52-d39f228fa43b\") " pod="openshift-ingress/payload-processing-5cc6cd444b-mmtk9" Jul 29 20:55:24.666307 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.666157 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372a884b-b391-42b9-9299-cb2173cbabe1-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-cdblw\" (UID: \"372a884b-b391-42b9-9299-cb2173cbabe1\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cdblw" Jul 29 20:55:24.666950 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.666896 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372a884b-b391-42b9-9299-cb2173cbabe1-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-cdblw\" (UID: \"372a884b-b391-42b9-9299-cb2173cbabe1\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cdblw" Jul 29 20:55:24.667146 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.667009 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24fdea1b-1074-47cf-9b52-d39f228fa43b-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-mmtk9\" (UID: \"24fdea1b-1074-47cf-9b52-d39f228fa43b\") " pod="openshift-ingress/payload-processing-5cc6cd444b-mmtk9" Jul 29 20:55:24.668720 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.668690 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34716149-c1ff-4198-b465-149d7c0e8d14-maas-api-tls\") pod \"maas-api-848d99f5fb-d85qg\" (UID: \"34716149-c1ff-4198-b465-149d7c0e8d14\") " pod="odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg" Jul 29 20:55:24.673676 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.673649 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bwkq\" (UniqueName: \"kubernetes.io/projected/372a884b-b391-42b9-9299-cb2173cbabe1-kube-api-access-7bwkq\") pod \"payload-pre-processing-687dbb6cf6-cdblw\" (UID: \"372a884b-b391-42b9-9299-cb2173cbabe1\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cdblw" Jul 29 20:55:24.674172 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.674150 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49zr6\" (UniqueName: \"kubernetes.io/projected/24fdea1b-1074-47cf-9b52-d39f228fa43b-kube-api-access-49zr6\") pod \"payload-processing-5cc6cd444b-mmtk9\" (UID: \"24fdea1b-1074-47cf-9b52-d39f228fa43b\") " pod="openshift-ingress/payload-processing-5cc6cd444b-mmtk9" Jul 29 20:55:24.794882 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.794788 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cdblw" Jul 29 20:55:24.821490 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.821456 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-mmtk9" Jul 29 20:55:24.869537 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.868986 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg" Jul 29 20:55:24.986833 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:24.986708 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-cdblw"] Jul 29 20:55:24.988288 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:55:24.988252 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372a884b_b391_42b9_9299_cb2173cbabe1.slice/crio-bbb70704c1c38b2365fcaada48bf4ad394fe750b95dbbad26450a420fb552081 WatchSource:0}: Error finding container bbb70704c1c38b2365fcaada48bf4ad394fe750b95dbbad26450a420fb552081: Status 404 returned error can't find the container with id bbb70704c1c38b2365fcaada48bf4ad394fe750b95dbbad26450a420fb552081 Jul 29 20:55:25.021262 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:25.021015 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-mmtk9"] Jul 29 20:55:25.090358 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:25.090330 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg"] Jul 29 20:55:25.093104 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:55:25.093076 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34716149_c1ff_4198_b465_149d7c0e8d14.slice/crio-bedafbade143f11323b9adb7ef10f0d7cbe1d509d854029c1d9b69bed54507aa WatchSource:0}: Error finding container bedafbade143f11323b9adb7ef10f0d7cbe1d509d854029c1d9b69bed54507aa: Status 404 returned error can't find the container with id bedafbade143f11323b9adb7ef10f0d7cbe1d509d854029c1d9b69bed54507aa Jul 29 20:55:25.615745 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:25.615695 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg" event={"ID":"34716149-c1ff-4198-b465-149d7c0e8d14","Type":"ContainerStarted","Data":"bedafbade143f11323b9adb7ef10f0d7cbe1d509d854029c1d9b69bed54507aa"} Jul 29 20:55:25.617509 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:25.617470 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cdblw" event={"ID":"372a884b-b391-42b9-9299-cb2173cbabe1","Type":"ContainerStarted","Data":"bbb70704c1c38b2365fcaada48bf4ad394fe750b95dbbad26450a420fb552081"} Jul 29 20:55:25.619389 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:25.619348 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-mmtk9" event={"ID":"24fdea1b-1074-47cf-9b52-d39f228fa43b","Type":"ContainerStarted","Data":"1196d2bac47f0edf90de3e905c268f1e1d1545fd9f80eabcdc97668e052bcb4c"} Jul 29 20:55:28.647423 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:28.647346 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg" event={"ID":"34716149-c1ff-4198-b465-149d7c0e8d14","Type":"ContainerStarted","Data":"1ffb30abb2249ed0c345f53114786f263af24d67e8d27a89be8c9d0e0c691aa6"} Jul 29 20:55:28.648112 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:28.647718 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg" Jul 29 20:55:28.654753 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:28.654251 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cdblw" event={"ID":"372a884b-b391-42b9-9299-cb2173cbabe1","Type":"ContainerStarted","Data":"2ef1a78be6f52e803b76c3b566c7d9bdc49c55e14967168f45632ae8787b2ae8"} Jul 29 20:55:28.654948 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:28.654908 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cdblw" Jul 29 20:55:28.657765 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:28.657733 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-mmtk9" event={"ID":"24fdea1b-1074-47cf-9b52-d39f228fa43b","Type":"ContainerStarted","Data":"722dc2ff930160bd110fa699e7a1f916621d22315353530374ff771c2aa14564"} Jul 29 20:55:28.657895 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:28.657874 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-mmtk9" Jul 29 20:55:28.664378 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:28.664332 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg" podStartSLOduration=2.923951442 podStartE2EDuration="5.664316891s" totalImagesPullingTime="2.740365449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:55:28.661827051 +0000 UTC m=+545.532930518" watchObservedRunningTime="2026-07-29 20:55:28.664316891 +0000 UTC m=+545.535420358" Jul 29 20:55:28.680404 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:28.680336 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-mmtk9" podStartSLOduration=1.860449054 podStartE2EDuration="4.680314877s" totalImagesPullingTime="2.819865823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:55:28.678146329 +0000 UTC m=+545.549249799" watchObservedRunningTime="2026-07-29 20:55:28.680314877 +0000 UTC m=+545.551418347" Jul 29 20:55:28.692865 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:28.692819 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cdblw" podStartSLOduration=1.833463014 podStartE2EDuration="4.692801721s" totalImagesPullingTime="2.859338707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:55:28.691983022 +0000 UTC m=+545.563086490" watchObservedRunningTime="2026-07-29 20:55:28.692801721 +0000 UTC m=+545.563905188" Jul 29 20:55:32.682736 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:32.682699 2562 generic.go:356] "Generic (PLEG): container finished" podID="fa5cd0c8-a7b7-45e0-9015-de4ad64b8219" containerID="16124d4257d1124e7e87e4200e887f978bf686c32daa643c6c4977a130052acb" exitCode=0 Jul 29 20:55:32.683216 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:32.682772 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" event={"ID":"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219","Type":"ContainerDied","Data":"16124d4257d1124e7e87e4200e887f978bf686c32daa643c6c4977a130052acb"} Jul 29 20:55:32.683216 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:32.683162 2562 scope.go:117] "RemoveContainer" containerID="16124d4257d1124e7e87e4200e887f978bf686c32daa643c6c4977a130052acb" Jul 29 20:55:32.684136 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:32.684110 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 20:55:32.684199 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:32.684180 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 20:55:32.684239 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:32.684214 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 20:55:33.714945 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:33.713972 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" event={"ID":"fa5cd0c8-a7b7-45e0-9015-de4ad64b8219","Type":"ContainerStarted","Data":"cc6fa252f1aa76c0c12d5211fe42f57c16f79eaaf1488794873a0e964b407157"} Jul 29 20:55:34.548296 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:34.548251 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:55:34.549567 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:34.549541 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2229g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 29 20:55:34.549697 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:34.549603 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" podUID="fa5cd0c8-a7b7-45e0-9015-de4ad64b8219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 29 20:55:34.669785 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:34.669754 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-848d99f5fb-d85qg" Jul 29 20:55:35.548653 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:35.548612 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2229g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 29 20:55:35.549126 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:35.548678 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" podUID="fa5cd0c8-a7b7-45e0-9015-de4ad64b8219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 29 20:55:36.548891 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:36.548846 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2229g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 29 20:55:36.549330 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:36.548909 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" podUID="fa5cd0c8-a7b7-45e0-9015-de4ad64b8219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 29 20:55:37.124934 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:37.124893 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:55:37.585445 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:37.585414 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:55:37.754489 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:37.754460 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2229g" Jul 29 20:55:39.665805 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:39.665768 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-mmtk9" Jul 29 20:55:39.666253 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:39.665879 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cdblw" Jul 29 20:55:46.640784 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.640746 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p"] Jul 29 20:55:46.644486 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.644465 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:46.646635 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.646607 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 29 20:55:46.646635 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.646626 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 29 20:55:46.647014 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.646998 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t86zw\"" Jul 29 20:55:46.647091 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.647037 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 29 20:55:46.653563 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.653194 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p"] Jul 29 20:55:46.729566 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.729532 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p\" (UID: \"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:46.729750 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.729574 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p\" (UID: \"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:46.729750 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.729607 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z6qt\" (UniqueName: \"kubernetes.io/projected/5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162-kube-api-access-8z6qt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p\" (UID: \"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:46.729750 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.729684 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p\" (UID: \"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:46.729750 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.729729 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p\" (UID: \"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:46.729892 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.729781 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p\" (UID: \"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:46.831224 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.831189 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p\" (UID: \"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:46.831224 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.831236 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p\" (UID: \"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:46.831506 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.831283 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p\" (UID: \"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:46.831506 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.831318 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p\" (UID: \"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:46.831506 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.831380 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z6qt\" (UniqueName: \"kubernetes.io/projected/5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162-kube-api-access-8z6qt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p\" (UID: \"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:46.831506 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.831447 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p\" (UID: \"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:46.831772 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.831739 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p\" (UID: \"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:46.831837 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.831785 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p\" (UID: \"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:46.831895 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.831866 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p\" (UID: \"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:46.833701 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.833679 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p\" (UID: \"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:46.834113 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.834093 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p\" (UID: \"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:46.838672 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.838649 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z6qt\" (UniqueName: \"kubernetes.io/projected/5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162-kube-api-access-8z6qt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p\" (UID: \"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:46.959363 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:46.959261 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:47.099242 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:47.099214 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p"] Jul 29 20:55:47.100012 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:55:47.099896 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd41d02_77ed_4d6b_9cb1_d6a5e5fda162.slice/crio-6bc141665747f841b4fe1c7abbf98d4ee9ef3586a44d9bb57144e58477eb35ae WatchSource:0}: Error finding container 6bc141665747f841b4fe1c7abbf98d4ee9ef3586a44d9bb57144e58477eb35ae: Status 404 returned error can't find the container with id 6bc141665747f841b4fe1c7abbf98d4ee9ef3586a44d9bb57144e58477eb35ae Jul 29 20:55:47.815122 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:47.815036 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" event={"ID":"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162","Type":"ContainerStarted","Data":"6bc141665747f841b4fe1c7abbf98d4ee9ef3586a44d9bb57144e58477eb35ae"} Jul 29 20:55:48.822209 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:48.822169 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" event={"ID":"5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162","Type":"ContainerStarted","Data":"08fdd32f04669b142b1ce04a5042abd56f077f3659a3eda7db8c38b5939e90af"} Jul 29 20:55:48.839583 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:48.839531 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" podStartSLOduration=1.675505311 podStartE2EDuration="2.839514591s" totalImagesPullingTime="1.16400928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:55:48.83655099 +0000 UTC m=+565.707654477" watchObservedRunningTime="2026-07-29 20:55:48.839514591 +0000 UTC m=+565.710618057" Jul 29 20:55:56.960334 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:56.960299 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:56.960991 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:56.960363 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:56.973043 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:56.973010 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:55:57.892956 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:55:57.892911 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p" Jul 29 20:56:23.518763 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:23.518727 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 20:56:23.519690 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:23.519668 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 20:56:37.544255 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.544217 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t"] Jul 29 20:56:37.549348 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.549328 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:37.552057 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.552033 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 29 20:56:37.559668 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.559646 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t"] Jul 29 20:56:37.616127 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.616090 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/862f377a-5b72-466b-8185-0cda71bbda3e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t\" (UID: \"862f377a-5b72-466b-8185-0cda71bbda3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:37.616291 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.616151 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/862f377a-5b72-466b-8185-0cda71bbda3e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t\" (UID: \"862f377a-5b72-466b-8185-0cda71bbda3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:37.616291 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.616173 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vb8v\" (UniqueName: \"kubernetes.io/projected/862f377a-5b72-466b-8185-0cda71bbda3e-kube-api-access-8vb8v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t\" (UID: \"862f377a-5b72-466b-8185-0cda71bbda3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:37.616291 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.616229 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/862f377a-5b72-466b-8185-0cda71bbda3e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t\" (UID: \"862f377a-5b72-466b-8185-0cda71bbda3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:37.616291 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.616277 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/862f377a-5b72-466b-8185-0cda71bbda3e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t\" (UID: \"862f377a-5b72-466b-8185-0cda71bbda3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:37.616441 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.616303 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/862f377a-5b72-466b-8185-0cda71bbda3e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t\" (UID: \"862f377a-5b72-466b-8185-0cda71bbda3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:37.717220 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.717183 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/862f377a-5b72-466b-8185-0cda71bbda3e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t\" (UID: \"862f377a-5b72-466b-8185-0cda71bbda3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:37.717402 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.717236 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/862f377a-5b72-466b-8185-0cda71bbda3e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t\" (UID: \"862f377a-5b72-466b-8185-0cda71bbda3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:37.717402 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.717266 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/862f377a-5b72-466b-8185-0cda71bbda3e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t\" (UID: \"862f377a-5b72-466b-8185-0cda71bbda3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:37.717402 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.717303 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/862f377a-5b72-466b-8185-0cda71bbda3e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t\" (UID: \"862f377a-5b72-466b-8185-0cda71bbda3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:37.717402 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.717361 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/862f377a-5b72-466b-8185-0cda71bbda3e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t\" (UID: \"862f377a-5b72-466b-8185-0cda71bbda3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:37.717402 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.717388 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vb8v\" (UniqueName: \"kubernetes.io/projected/862f377a-5b72-466b-8185-0cda71bbda3e-kube-api-access-8vb8v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t\" (UID: \"862f377a-5b72-466b-8185-0cda71bbda3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:37.717692 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.717661 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/862f377a-5b72-466b-8185-0cda71bbda3e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t\" (UID: \"862f377a-5b72-466b-8185-0cda71bbda3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:37.717750 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.717669 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/862f377a-5b72-466b-8185-0cda71bbda3e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t\" (UID: \"862f377a-5b72-466b-8185-0cda71bbda3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:37.718036 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.718011 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/862f377a-5b72-466b-8185-0cda71bbda3e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t\" (UID: \"862f377a-5b72-466b-8185-0cda71bbda3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:37.720272 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.720248 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/862f377a-5b72-466b-8185-0cda71bbda3e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t\" (UID: \"862f377a-5b72-466b-8185-0cda71bbda3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:37.720606 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.720577 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/862f377a-5b72-466b-8185-0cda71bbda3e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t\" (UID: \"862f377a-5b72-466b-8185-0cda71bbda3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:37.725066 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.725035 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vb8v\" (UniqueName: \"kubernetes.io/projected/862f377a-5b72-466b-8185-0cda71bbda3e-kube-api-access-8vb8v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t\" (UID: \"862f377a-5b72-466b-8185-0cda71bbda3e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:37.863157 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:37.863124 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:38.001953 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:38.001911 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t"] Jul 29 20:56:38.003182 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:56:38.003151 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862f377a_5b72_466b_8185_0cda71bbda3e.slice/crio-e41272dc4bac4cbb345578c749533de25a6e52d013d87d05a92dab9e872871f1 WatchSource:0}: Error finding container e41272dc4bac4cbb345578c749533de25a6e52d013d87d05a92dab9e872871f1: Status 404 returned error can't find the container with id e41272dc4bac4cbb345578c749533de25a6e52d013d87d05a92dab9e872871f1 Jul 29 20:56:38.004576 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:38.004555 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 20:56:38.112348 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:38.112310 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" event={"ID":"862f377a-5b72-466b-8185-0cda71bbda3e","Type":"ContainerStarted","Data":"e41272dc4bac4cbb345578c749533de25a6e52d013d87d05a92dab9e872871f1"} Jul 29 20:56:39.123014 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:39.122977 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" event={"ID":"862f377a-5b72-466b-8185-0cda71bbda3e","Type":"ContainerStarted","Data":"71c23aafc19b9ae61f1deb6791077d6d88ddd59c0b03e0d22fb809ca741c4edc"} Jul 29 20:56:39.141170 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:39.141111 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" podStartSLOduration=1.981005658 podStartE2EDuration="2.141094246s" totalImagesPullingTime="160.088588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:56:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:56:39.139199222 +0000 UTC m=+616.010302689" watchObservedRunningTime="2026-07-29 20:56:39.141094246 +0000 UTC m=+616.012197758" Jul 29 20:56:47.863626 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:47.863530 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:47.863626 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:47.863606 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:47.876564 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:47.876536 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:48.186197 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:48.186108 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t" Jul 29 20:56:52.529977 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.529939 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw"] Jul 29 20:56:52.533707 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.533689 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.535789 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.535765 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 29 20:56:52.542493 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.542467 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw"] Jul 29 20:56:52.660424 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.660390 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f9f2a65-b3f3-4b62-a674-0d305110b511-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdkhw\" (UID: \"3f9f2a65-b3f3-4b62-a674-0d305110b511\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.660634 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.660439 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4kjb\" (UniqueName: \"kubernetes.io/projected/3f9f2a65-b3f3-4b62-a674-0d305110b511-kube-api-access-x4kjb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdkhw\" (UID: \"3f9f2a65-b3f3-4b62-a674-0d305110b511\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.660634 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.660475 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f9f2a65-b3f3-4b62-a674-0d305110b511-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdkhw\" (UID: \"3f9f2a65-b3f3-4b62-a674-0d305110b511\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.660634 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.660561 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f9f2a65-b3f3-4b62-a674-0d305110b511-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdkhw\" (UID: \"3f9f2a65-b3f3-4b62-a674-0d305110b511\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.660634 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.660597 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f9f2a65-b3f3-4b62-a674-0d305110b511-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdkhw\" (UID: \"3f9f2a65-b3f3-4b62-a674-0d305110b511\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.660834 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.660762 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f9f2a65-b3f3-4b62-a674-0d305110b511-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdkhw\" (UID: \"3f9f2a65-b3f3-4b62-a674-0d305110b511\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.762421 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.762388 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f9f2a65-b3f3-4b62-a674-0d305110b511-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdkhw\" (UID: \"3f9f2a65-b3f3-4b62-a674-0d305110b511\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.762627 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.762428 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4kjb\" (UniqueName: \"kubernetes.io/projected/3f9f2a65-b3f3-4b62-a674-0d305110b511-kube-api-access-x4kjb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdkhw\" (UID: \"3f9f2a65-b3f3-4b62-a674-0d305110b511\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.762627 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.762451 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f9f2a65-b3f3-4b62-a674-0d305110b511-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdkhw\" (UID: \"3f9f2a65-b3f3-4b62-a674-0d305110b511\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.762627 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.762490 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f9f2a65-b3f3-4b62-a674-0d305110b511-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdkhw\" (UID: \"3f9f2a65-b3f3-4b62-a674-0d305110b511\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.762627 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.762507 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f9f2a65-b3f3-4b62-a674-0d305110b511-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdkhw\" (UID: \"3f9f2a65-b3f3-4b62-a674-0d305110b511\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.762627 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.762563 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f9f2a65-b3f3-4b62-a674-0d305110b511-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdkhw\" (UID: \"3f9f2a65-b3f3-4b62-a674-0d305110b511\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.763029 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.763006 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f9f2a65-b3f3-4b62-a674-0d305110b511-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdkhw\" (UID: \"3f9f2a65-b3f3-4b62-a674-0d305110b511\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.763029 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.763021 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f9f2a65-b3f3-4b62-a674-0d305110b511-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdkhw\" (UID: \"3f9f2a65-b3f3-4b62-a674-0d305110b511\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.763175 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.763051 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f9f2a65-b3f3-4b62-a674-0d305110b511-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdkhw\" (UID: \"3f9f2a65-b3f3-4b62-a674-0d305110b511\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.764752 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.764729 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f9f2a65-b3f3-4b62-a674-0d305110b511-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdkhw\" (UID: \"3f9f2a65-b3f3-4b62-a674-0d305110b511\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.765043 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.765021 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f9f2a65-b3f3-4b62-a674-0d305110b511-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdkhw\" (UID: \"3f9f2a65-b3f3-4b62-a674-0d305110b511\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.769817 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.769795 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4kjb\" (UniqueName: \"kubernetes.io/projected/3f9f2a65-b3f3-4b62-a674-0d305110b511-kube-api-access-x4kjb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdkhw\" (UID: \"3f9f2a65-b3f3-4b62-a674-0d305110b511\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.846387 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.846346 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:56:52.990098 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:52.990067 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw"] Jul 29 20:56:52.991454 ip-10-0-137-216 kubenswrapper[2562]: W0729 20:56:52.991427 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9f2a65_b3f3_4b62_a674_0d305110b511.slice/crio-5909088e41be9a8d3cbb18565c5519075073835ebcdf9698f254e1a0306f3031 WatchSource:0}: Error finding container 5909088e41be9a8d3cbb18565c5519075073835ebcdf9698f254e1a0306f3031: Status 404 returned error can't find the container with id 5909088e41be9a8d3cbb18565c5519075073835ebcdf9698f254e1a0306f3031 Jul 29 20:56:53.204988 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:53.204949 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" event={"ID":"3f9f2a65-b3f3-4b62-a674-0d305110b511","Type":"ContainerStarted","Data":"5909088e41be9a8d3cbb18565c5519075073835ebcdf9698f254e1a0306f3031"} Jul 29 20:56:54.213015 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:54.212974 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" event={"ID":"3f9f2a65-b3f3-4b62-a674-0d305110b511","Type":"ContainerStarted","Data":"5a2d7bd9680df05804d366a80b8a2612f578906f04f13618e0cb47548ec3d4ea"} Jul 29 20:56:54.230221 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:56:54.230163 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" podStartSLOduration=2.048588562 podStartE2EDuration="2.230145982s" totalImagesPullingTime="181.55742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:56:54.228939924 +0000 UTC m=+631.100043389" watchObservedRunningTime="2026-07-29 20:56:54.230145982 +0000 UTC m=+631.101249460" Jul 29 20:57:02.846640 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:57:02.846595 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:57:02.847083 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:57:02.846669 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:57:02.859885 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:57:02.859853 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:57:03.281786 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:57:03.281753 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdkhw" Jul 29 20:58:44.405571 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:58:44.405531 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c85847677-qgrnz"] Jul 29 20:58:44.406195 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:58:44.405791 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5c85847677-qgrnz" podUID="f605d20b-2271-47c8-b61e-9a5a70e51121" containerName="manager" containerID="cri-o://b4766c482ffd12e0769c8c9a4dcaceacfb724f2a1651b66b4433c75815edc63f" gracePeriod=30 Jul 29 20:58:44.670732 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:58:44.670651 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c85847677-qgrnz" Jul 29 20:58:44.779675 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:58:44.779633 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f605d20b-2271-47c8-b61e-9a5a70e51121-webhook-cert\") pod \"f605d20b-2271-47c8-b61e-9a5a70e51121\" (UID: \"f605d20b-2271-47c8-b61e-9a5a70e51121\") " Jul 29 20:58:44.779863 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:58:44.779817 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr7qd\" (UniqueName: \"kubernetes.io/projected/f605d20b-2271-47c8-b61e-9a5a70e51121-kube-api-access-qr7qd\") pod \"f605d20b-2271-47c8-b61e-9a5a70e51121\" (UID: \"f605d20b-2271-47c8-b61e-9a5a70e51121\") " Jul 29 20:58:44.781932 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:58:44.781893 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f605d20b-2271-47c8-b61e-9a5a70e51121-kube-api-access-qr7qd" (OuterVolumeSpecName: "kube-api-access-qr7qd") pod "f605d20b-2271-47c8-b61e-9a5a70e51121" (UID: "f605d20b-2271-47c8-b61e-9a5a70e51121"). InnerVolumeSpecName "kube-api-access-qr7qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 20:58:44.782035 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:58:44.781932 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f605d20b-2271-47c8-b61e-9a5a70e51121-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f605d20b-2271-47c8-b61e-9a5a70e51121" (UID: "f605d20b-2271-47c8-b61e-9a5a70e51121"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 20:58:44.878440 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:58:44.878403 2562 generic.go:356] "Generic (PLEG): container finished" podID="f605d20b-2271-47c8-b61e-9a5a70e51121" containerID="b4766c482ffd12e0769c8c9a4dcaceacfb724f2a1651b66b4433c75815edc63f" exitCode=0 Jul 29 20:58:44.878440 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:58:44.878442 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c85847677-qgrnz" event={"ID":"f605d20b-2271-47c8-b61e-9a5a70e51121","Type":"ContainerDied","Data":"b4766c482ffd12e0769c8c9a4dcaceacfb724f2a1651b66b4433c75815edc63f"} Jul 29 20:58:44.878703 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:58:44.878453 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c85847677-qgrnz" Jul 29 20:58:44.878703 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:58:44.878484 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c85847677-qgrnz" event={"ID":"f605d20b-2271-47c8-b61e-9a5a70e51121","Type":"ContainerDied","Data":"01d9cc4a10e0982b5502cef363240a7d42e69921a0eac39e3bc6cb7b2535c7a6"} Jul 29 20:58:44.878703 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:58:44.878505 2562 scope.go:117] "RemoveContainer" containerID="b4766c482ffd12e0769c8c9a4dcaceacfb724f2a1651b66b4433c75815edc63f" Jul 29 20:58:44.880913 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:58:44.880890 2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f605d20b-2271-47c8-b61e-9a5a70e51121-webhook-cert\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:58:44.881068 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:58:44.880938 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr7qd\" (UniqueName: \"kubernetes.io/projected/f605d20b-2271-47c8-b61e-9a5a70e51121-kube-api-access-qr7qd\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 20:58:44.888168 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:58:44.888149 2562 scope.go:117] "RemoveContainer" containerID="b4766c482ffd12e0769c8c9a4dcaceacfb724f2a1651b66b4433c75815edc63f" Jul 29 20:58:44.888414 ip-10-0-137-216 kubenswrapper[2562]: E0729 20:58:44.888394 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4766c482ffd12e0769c8c9a4dcaceacfb724f2a1651b66b4433c75815edc63f\": container with ID starting with b4766c482ffd12e0769c8c9a4dcaceacfb724f2a1651b66b4433c75815edc63f not found: ID does not exist" containerID="b4766c482ffd12e0769c8c9a4dcaceacfb724f2a1651b66b4433c75815edc63f" Jul 29 20:58:44.888466 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:58:44.888424 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4766c482ffd12e0769c8c9a4dcaceacfb724f2a1651b66b4433c75815edc63f"} err="failed to get container status \"b4766c482ffd12e0769c8c9a4dcaceacfb724f2a1651b66b4433c75815edc63f\": rpc error: code = NotFound desc = could not find container \"b4766c482ffd12e0769c8c9a4dcaceacfb724f2a1651b66b4433c75815edc63f\": container with ID starting with b4766c482ffd12e0769c8c9a4dcaceacfb724f2a1651b66b4433c75815edc63f not found: ID does not exist" Jul 29 20:58:44.901535 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:58:44.901494 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c85847677-qgrnz"] Jul 29 20:58:44.905327 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:58:44.905298 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c85847677-qgrnz"] Jul 29 20:58:45.558432 ip-10-0-137-216 kubenswrapper[2562]: I0729 20:58:45.558399 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f605d20b-2271-47c8-b61e-9a5a70e51121" path="/var/lib/kubelet/pods/f605d20b-2271-47c8-b61e-9a5a70e51121/volumes" Jul 29 21:01:23.560390 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:01:23.560314 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:01:23.563954 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:01:23.563910 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:06:23.610072 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:06:23.610045 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:06:23.615273 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:06:23.615246 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:07:14.390303 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:07:14.390261 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch"] Jul 29 21:07:14.391004 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:07:14.390494 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch" podUID="ee5e5207-ca36-433e-9d3d-0fb90c066122" containerName="manager" containerID="cri-o://a67bd65284ee12d88108f22575341b20b5b5df50c84c81eafb82b555beae0c53" gracePeriod=10 Jul 29 21:07:14.653250 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:07:14.653188 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch" Jul 29 21:07:14.725105 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:07:14.725073 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fm9pl\" (UniqueName: \"kubernetes.io/projected/ee5e5207-ca36-433e-9d3d-0fb90c066122-kube-api-access-fm9pl\") pod \"ee5e5207-ca36-433e-9d3d-0fb90c066122\" (UID: \"ee5e5207-ca36-433e-9d3d-0fb90c066122\") " Jul 29 21:07:14.725283 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:07:14.725156 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee5e5207-ca36-433e-9d3d-0fb90c066122-extensions-socket-volume\") pod \"ee5e5207-ca36-433e-9d3d-0fb90c066122\" (UID: \"ee5e5207-ca36-433e-9d3d-0fb90c066122\") " Jul 29 21:07:14.725537 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:07:14.725510 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5e5207-ca36-433e-9d3d-0fb90c066122-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ee5e5207-ca36-433e-9d3d-0fb90c066122" (UID: "ee5e5207-ca36-433e-9d3d-0fb90c066122"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:07:14.727163 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:07:14.727141 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5e5207-ca36-433e-9d3d-0fb90c066122-kube-api-access-fm9pl" (OuterVolumeSpecName: "kube-api-access-fm9pl") pod "ee5e5207-ca36-433e-9d3d-0fb90c066122" (UID: "ee5e5207-ca36-433e-9d3d-0fb90c066122"). InnerVolumeSpecName "kube-api-access-fm9pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:07:14.826410 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:07:14.826372 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm9pl\" (UniqueName: \"kubernetes.io/projected/ee5e5207-ca36-433e-9d3d-0fb90c066122-kube-api-access-fm9pl\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:07:14.826410 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:07:14.826404 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee5e5207-ca36-433e-9d3d-0fb90c066122-extensions-socket-volume\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:07:14.951360 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:07:14.951271 2562 generic.go:356] "Generic (PLEG): container finished" podID="ee5e5207-ca36-433e-9d3d-0fb90c066122" containerID="a67bd65284ee12d88108f22575341b20b5b5df50c84c81eafb82b555beae0c53" exitCode=0 Jul 29 21:07:14.951360 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:07:14.951310 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch" Jul 29 21:07:14.951575 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:07:14.951356 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch" event={"ID":"ee5e5207-ca36-433e-9d3d-0fb90c066122","Type":"ContainerDied","Data":"a67bd65284ee12d88108f22575341b20b5b5df50c84c81eafb82b555beae0c53"} Jul 29 21:07:14.951575 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:07:14.951409 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch" event={"ID":"ee5e5207-ca36-433e-9d3d-0fb90c066122","Type":"ContainerDied","Data":"e4bfd9b026a50bd5150b630f886c9d2fbc52f6683db314f5730bb74a0a3e570c"} Jul 29 21:07:14.951575 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:07:14.951436 2562 scope.go:117] "RemoveContainer" containerID="a67bd65284ee12d88108f22575341b20b5b5df50c84c81eafb82b555beae0c53" Jul 29 21:07:14.962504 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:07:14.962482 2562 scope.go:117] "RemoveContainer" containerID="a67bd65284ee12d88108f22575341b20b5b5df50c84c81eafb82b555beae0c53" Jul 29 21:07:14.962761 ip-10-0-137-216 kubenswrapper[2562]: E0729 21:07:14.962742 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a67bd65284ee12d88108f22575341b20b5b5df50c84c81eafb82b555beae0c53\": container with ID starting with a67bd65284ee12d88108f22575341b20b5b5df50c84c81eafb82b555beae0c53 not found: ID does not exist" containerID="a67bd65284ee12d88108f22575341b20b5b5df50c84c81eafb82b555beae0c53" Jul 29 21:07:14.962801 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:07:14.962770 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a67bd65284ee12d88108f22575341b20b5b5df50c84c81eafb82b555beae0c53"} err="failed to get container status \"a67bd65284ee12d88108f22575341b20b5b5df50c84c81eafb82b555beae0c53\": rpc error: code = NotFound desc = could not find container \"a67bd65284ee12d88108f22575341b20b5b5df50c84c81eafb82b555beae0c53\": container with ID starting with a67bd65284ee12d88108f22575341b20b5b5df50c84c81eafb82b555beae0c53 not found: ID does not exist" Jul 29 21:07:14.974953 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:07:14.974874 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch"] Jul 29 21:07:14.978330 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:07:14.978307 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xfmch"] Jul 29 21:07:15.557739 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:07:15.557658 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee5e5207-ca36-433e-9d3d-0fb90c066122" path="/var/lib/kubelet/pods/ee5e5207-ca36-433e-9d3d-0fb90c066122/volumes" Jul 29 21:08:15.096122 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:15.096079 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj"] Jul 29 21:08:15.096623 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:15.096514 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f605d20b-2271-47c8-b61e-9a5a70e51121" containerName="manager" Jul 29 21:08:15.096623 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:15.096526 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f605d20b-2271-47c8-b61e-9a5a70e51121" containerName="manager" Jul 29 21:08:15.096623 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:15.096545 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee5e5207-ca36-433e-9d3d-0fb90c066122" containerName="manager" Jul 29 21:08:15.096623 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:15.096551 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee5e5207-ca36-433e-9d3d-0fb90c066122" containerName="manager" Jul 29 21:08:15.096623 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:15.096613 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee5e5207-ca36-433e-9d3d-0fb90c066122" containerName="manager" Jul 29 21:08:15.096623 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:15.096623 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f605d20b-2271-47c8-b61e-9a5a70e51121" containerName="manager" Jul 29 21:08:15.099789 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:15.099772 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj" Jul 29 21:08:15.101772 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:15.101752 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mbgcs\"" Jul 29 21:08:15.110572 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:15.110531 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj"] Jul 29 21:08:15.241907 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:15.241866 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b45ed17c-c04e-42af-9ff8-4ae865e3ed50-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gmcmj\" (UID: \"b45ed17c-c04e-42af-9ff8-4ae865e3ed50\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj" Jul 29 21:08:15.242118 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:15.241952 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75cpt\" (UniqueName: \"kubernetes.io/projected/b45ed17c-c04e-42af-9ff8-4ae865e3ed50-kube-api-access-75cpt\") pod \"kuadrant-operator-controller-manager-86878c769f-gmcmj\" (UID: \"b45ed17c-c04e-42af-9ff8-4ae865e3ed50\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj" Jul 29 21:08:15.342771 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:15.342733 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75cpt\" (UniqueName: \"kubernetes.io/projected/b45ed17c-c04e-42af-9ff8-4ae865e3ed50-kube-api-access-75cpt\") pod \"kuadrant-operator-controller-manager-86878c769f-gmcmj\" (UID: \"b45ed17c-c04e-42af-9ff8-4ae865e3ed50\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj" Jul 29 21:08:15.342996 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:15.342868 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b45ed17c-c04e-42af-9ff8-4ae865e3ed50-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gmcmj\" (UID: \"b45ed17c-c04e-42af-9ff8-4ae865e3ed50\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj" Jul 29 21:08:15.343234 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:15.343215 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b45ed17c-c04e-42af-9ff8-4ae865e3ed50-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gmcmj\" (UID: \"b45ed17c-c04e-42af-9ff8-4ae865e3ed50\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj" Jul 29 21:08:15.351394 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:15.351326 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75cpt\" (UniqueName: \"kubernetes.io/projected/b45ed17c-c04e-42af-9ff8-4ae865e3ed50-kube-api-access-75cpt\") pod \"kuadrant-operator-controller-manager-86878c769f-gmcmj\" (UID: \"b45ed17c-c04e-42af-9ff8-4ae865e3ed50\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj" Jul 29 21:08:15.411368 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:15.411324 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj" Jul 29 21:08:15.558271 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:15.558245 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj"] Jul 29 21:08:15.558969 ip-10-0-137-216 kubenswrapper[2562]: W0729 21:08:15.558906 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45ed17c_c04e_42af_9ff8_4ae865e3ed50.slice/crio-b3b7fbdf2d9ad4d4a68dbfd7e9b9443b76d9b3caaac9710c53122f73de43babc WatchSource:0}: Error finding container b3b7fbdf2d9ad4d4a68dbfd7e9b9443b76d9b3caaac9710c53122f73de43babc: Status 404 returned error can't find the container with id b3b7fbdf2d9ad4d4a68dbfd7e9b9443b76d9b3caaac9710c53122f73de43babc Jul 29 21:08:16.308726 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:16.308691 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj" event={"ID":"b45ed17c-c04e-42af-9ff8-4ae865e3ed50","Type":"ContainerStarted","Data":"5166a21211e941b6ebbc71a48ff77b6f8fe1456931c67bb9ac7f7ece2379d18c"} Jul 29 21:08:16.309152 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:16.308751 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj" event={"ID":"b45ed17c-c04e-42af-9ff8-4ae865e3ed50","Type":"ContainerStarted","Data":"b3b7fbdf2d9ad4d4a68dbfd7e9b9443b76d9b3caaac9710c53122f73de43babc"} Jul 29 21:08:16.309152 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:16.308944 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj" Jul 29 21:08:16.331144 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:16.331092 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj" podStartSLOduration=1.33107744 podStartE2EDuration="1.33107744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:08:16.328367573 +0000 UTC m=+1313.199471041" watchObservedRunningTime="2026-07-29 21:08:16.33107744 +0000 UTC m=+1313.202180904" Jul 29 21:08:27.317416 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:08:27.317376 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj" Jul 29 21:11:23.650274 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:11:23.650163 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:11:23.657877 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:11:23.657855 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:16:23.690357 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:23.690244 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:16:23.701120 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:23.701087 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:16:26.739705 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:26.739664 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb"] Jul 29 21:16:26.743985 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:26.743960 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" Jul 29 21:16:26.745906 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:26.745886 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-23c87980-serving-cert\"" Jul 29 21:16:26.753692 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:26.753626 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb"] Jul 29 21:16:26.808616 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:26.808578 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feef996d-af6e-4cf9-9f3c-a1790cce43ed-maas-api-tls\") pod \"maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb\" (UID: \"feef996d-af6e-4cf9-9f3c-a1790cce43ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" Jul 29 21:16:26.808616 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:26.808626 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx94n\" (UniqueName: \"kubernetes.io/projected/feef996d-af6e-4cf9-9f3c-a1790cce43ed-kube-api-access-hx94n\") pod \"maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb\" (UID: \"feef996d-af6e-4cf9-9f3c-a1790cce43ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" Jul 29 21:16:26.808851 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:26.808653 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feef996d-af6e-4cf9-9f3c-a1790cce43ed-openshift-service-ca\") pod \"maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb\" (UID: \"feef996d-af6e-4cf9-9f3c-a1790cce43ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" Jul 29 21:16:26.910246 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:26.910208 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feef996d-af6e-4cf9-9f3c-a1790cce43ed-maas-api-tls\") pod \"maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb\" (UID: \"feef996d-af6e-4cf9-9f3c-a1790cce43ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" Jul 29 21:16:26.910451 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:26.910250 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx94n\" (UniqueName: \"kubernetes.io/projected/feef996d-af6e-4cf9-9f3c-a1790cce43ed-kube-api-access-hx94n\") pod \"maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb\" (UID: \"feef996d-af6e-4cf9-9f3c-a1790cce43ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" Jul 29 21:16:26.910451 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:26.910284 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feef996d-af6e-4cf9-9f3c-a1790cce43ed-openshift-service-ca\") pod \"maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb\" (UID: \"feef996d-af6e-4cf9-9f3c-a1790cce43ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" Jul 29 21:16:26.911046 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:26.911017 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feef996d-af6e-4cf9-9f3c-a1790cce43ed-openshift-service-ca\") pod \"maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb\" (UID: \"feef996d-af6e-4cf9-9f3c-a1790cce43ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" Jul 29 21:16:26.912734 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:26.912709 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feef996d-af6e-4cf9-9f3c-a1790cce43ed-maas-api-tls\") pod \"maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb\" (UID: \"feef996d-af6e-4cf9-9f3c-a1790cce43ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" Jul 29 21:16:26.917790 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:26.917767 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx94n\" (UniqueName: \"kubernetes.io/projected/feef996d-af6e-4cf9-9f3c-a1790cce43ed-kube-api-access-hx94n\") pod \"maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb\" (UID: \"feef996d-af6e-4cf9-9f3c-a1790cce43ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" Jul 29 21:16:27.055521 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:27.055488 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" Jul 29 21:16:27.207163 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:27.207127 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb"] Jul 29 21:16:27.207536 ip-10-0-137-216 kubenswrapper[2562]: W0729 21:16:27.207510 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeef996d_af6e_4cf9_9f3c_a1790cce43ed.slice/crio-97ab33e396efe547f62ac4f7c6e574ffe0759055edd71d5d6e50848f8c70c393 WatchSource:0}: Error finding container 97ab33e396efe547f62ac4f7c6e574ffe0759055edd71d5d6e50848f8c70c393: Status 404 returned error can't find the container with id 97ab33e396efe547f62ac4f7c6e574ffe0759055edd71d5d6e50848f8c70c393 Jul 29 21:16:27.209123 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:27.209101 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 21:16:27.274002 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:27.273951 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" event={"ID":"feef996d-af6e-4cf9-9f3c-a1790cce43ed","Type":"ContainerStarted","Data":"97ab33e396efe547f62ac4f7c6e574ffe0759055edd71d5d6e50848f8c70c393"} Jul 29 21:16:27.385093 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:27.384992 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9"] Jul 29 21:16:27.389876 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:27.389852 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9" Jul 29 21:16:27.391692 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:27.391668 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-23c87980-dockercfg-qmv58\"" Jul 29 21:16:27.391822 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:27.391715 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-23c87980\"" Jul 29 21:16:27.401525 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:27.401494 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9"] Jul 29 21:16:27.517587 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:27.517552 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgpkh\" (UniqueName: \"kubernetes.io/projected/248b1556-db55-4071-8afd-3f350e0d10b5-kube-api-access-dgpkh\") pod \"payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9\" (UID: \"248b1556-db55-4071-8afd-3f350e0d10b5\") " pod="openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9" Jul 29 21:16:27.517760 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:27.517617 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248b1556-db55-4071-8afd-3f350e0d10b5-plugins-config-volume\") pod \"payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9\" (UID: \"248b1556-db55-4071-8afd-3f350e0d10b5\") " pod="openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9" Jul 29 21:16:27.618810 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:27.618777 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgpkh\" (UniqueName: \"kubernetes.io/projected/248b1556-db55-4071-8afd-3f350e0d10b5-kube-api-access-dgpkh\") pod \"payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9\" (UID: \"248b1556-db55-4071-8afd-3f350e0d10b5\") " pod="openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9" Jul 29 21:16:27.619042 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:27.618834 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248b1556-db55-4071-8afd-3f350e0d10b5-plugins-config-volume\") pod \"payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9\" (UID: \"248b1556-db55-4071-8afd-3f350e0d10b5\") " pod="openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9" Jul 29 21:16:27.619558 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:27.619536 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248b1556-db55-4071-8afd-3f350e0d10b5-plugins-config-volume\") pod \"payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9\" (UID: \"248b1556-db55-4071-8afd-3f350e0d10b5\") " pod="openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9" Jul 29 21:16:27.626796 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:27.626776 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgpkh\" (UniqueName: \"kubernetes.io/projected/248b1556-db55-4071-8afd-3f350e0d10b5-kube-api-access-dgpkh\") pod \"payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9\" (UID: \"248b1556-db55-4071-8afd-3f350e0d10b5\") " pod="openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9" Jul 29 21:16:27.703731 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:27.703627 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9" Jul 29 21:16:27.851487 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:27.851458 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9"] Jul 29 21:16:27.853456 ip-10-0-137-216 kubenswrapper[2562]: W0729 21:16:27.853427 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248b1556_db55_4071_8afd_3f350e0d10b5.slice/crio-a4ac6470b6d278895dd2207cf1a61873a7690928cff03076053eba43eeb6a558 WatchSource:0}: Error finding container a4ac6470b6d278895dd2207cf1a61873a7690928cff03076053eba43eeb6a558: Status 404 returned error can't find the container with id a4ac6470b6d278895dd2207cf1a61873a7690928cff03076053eba43eeb6a558 Jul 29 21:16:28.282155 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:28.282115 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9" event={"ID":"248b1556-db55-4071-8afd-3f350e0d10b5","Type":"ContainerStarted","Data":"ad7ab4ad20f6ad1228565a1ca64f788688c59cab5c1a41ae18f34e869479cff3"} Jul 29 21:16:28.282345 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:28.282182 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9" event={"ID":"248b1556-db55-4071-8afd-3f350e0d10b5","Type":"ContainerStarted","Data":"a4ac6470b6d278895dd2207cf1a61873a7690928cff03076053eba43eeb6a558"} Jul 29 21:16:28.282345 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:28.282267 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9" Jul 29 21:16:28.283832 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:28.283804 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" event={"ID":"feef996d-af6e-4cf9-9f3c-a1790cce43ed","Type":"ContainerStarted","Data":"4f249db4cbadf6dc8146bcbcdcb6f5253d7c30047c2742c7360d3a5a20c3dce6"} Jul 29 21:16:28.283975 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:28.283858 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" Jul 29 21:16:28.301019 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:28.300962 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9" podStartSLOduration=1.300941947 podStartE2EDuration="1.300941947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:28.296431402 +0000 UTC m=+1805.167534870" watchObservedRunningTime="2026-07-29 21:16:28.300941947 +0000 UTC m=+1805.172045411" Jul 29 21:16:28.314335 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:28.314284 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" podStartSLOduration=1.710589656 podStartE2EDuration="2.314265726s" totalImagesPullingTime="603.67607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:28.309354204 +0000 UTC m=+1805.180457672" watchObservedRunningTime="2026-07-29 21:16:28.314265726 +0000 UTC m=+1805.185369192" Jul 29 21:16:34.295584 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:34.295548 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" Jul 29 21:16:39.292451 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:39.292407 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9" Jul 29 21:16:57.190418 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.190383 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb"] Jul 29 21:16:57.191052 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.190642 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" podUID="feef996d-af6e-4cf9-9f3c-a1790cce43ed" containerName="maas-api" containerID="cri-o://4f249db4cbadf6dc8146bcbcdcb6f5253d7c30047c2742c7360d3a5a20c3dce6" gracePeriod=30 Jul 29 21:16:57.248180 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.248141 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9"] Jul 29 21:16:57.248612 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.248420 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9" podUID="248b1556-db55-4071-8afd-3f350e0d10b5" containerName="payload-processing" containerID="cri-o://ad7ab4ad20f6ad1228565a1ca64f788688c59cab5c1a41ae18f34e869479cff3" gracePeriod=30 Jul 29 21:16:57.477951 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.477888 2562 generic.go:356] "Generic (PLEG): container finished" podID="feef996d-af6e-4cf9-9f3c-a1790cce43ed" containerID="4f249db4cbadf6dc8146bcbcdcb6f5253d7c30047c2742c7360d3a5a20c3dce6" exitCode=0 Jul 29 21:16:57.478122 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.477968 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" event={"ID":"feef996d-af6e-4cf9-9f3c-a1790cce43ed","Type":"ContainerDied","Data":"4f249db4cbadf6dc8146bcbcdcb6f5253d7c30047c2742c7360d3a5a20c3dce6"} Jul 29 21:16:57.480134 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.480103 2562 generic.go:356] "Generic (PLEG): container finished" podID="248b1556-db55-4071-8afd-3f350e0d10b5" containerID="ad7ab4ad20f6ad1228565a1ca64f788688c59cab5c1a41ae18f34e869479cff3" exitCode=0 Jul 29 21:16:57.480278 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.480176 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9" event={"ID":"248b1556-db55-4071-8afd-3f350e0d10b5","Type":"ContainerDied","Data":"ad7ab4ad20f6ad1228565a1ca64f788688c59cab5c1a41ae18f34e869479cff3"} Jul 29 21:16:57.515689 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.515660 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" Jul 29 21:16:57.550389 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.550361 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9" Jul 29 21:16:57.619135 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.619091 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgpkh\" (UniqueName: \"kubernetes.io/projected/248b1556-db55-4071-8afd-3f350e0d10b5-kube-api-access-dgpkh\") pod \"248b1556-db55-4071-8afd-3f350e0d10b5\" (UID: \"248b1556-db55-4071-8afd-3f350e0d10b5\") " Jul 29 21:16:57.619135 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.619141 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feef996d-af6e-4cf9-9f3c-a1790cce43ed-openshift-service-ca\") pod \"feef996d-af6e-4cf9-9f3c-a1790cce43ed\" (UID: \"feef996d-af6e-4cf9-9f3c-a1790cce43ed\") " Jul 29 21:16:57.619405 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.619204 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248b1556-db55-4071-8afd-3f350e0d10b5-plugins-config-volume\") pod \"248b1556-db55-4071-8afd-3f350e0d10b5\" (UID: \"248b1556-db55-4071-8afd-3f350e0d10b5\") " Jul 29 21:16:57.619405 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.619228 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx94n\" (UniqueName: \"kubernetes.io/projected/feef996d-af6e-4cf9-9f3c-a1790cce43ed-kube-api-access-hx94n\") pod \"feef996d-af6e-4cf9-9f3c-a1790cce43ed\" (UID: \"feef996d-af6e-4cf9-9f3c-a1790cce43ed\") " Jul 29 21:16:57.619405 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.619273 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feef996d-af6e-4cf9-9f3c-a1790cce43ed-maas-api-tls\") pod \"feef996d-af6e-4cf9-9f3c-a1790cce43ed\" (UID: \"feef996d-af6e-4cf9-9f3c-a1790cce43ed\") " Jul 29 21:16:57.619753 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.619720 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feef996d-af6e-4cf9-9f3c-a1790cce43ed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "feef996d-af6e-4cf9-9f3c-a1790cce43ed" (UID: "feef996d-af6e-4cf9-9f3c-a1790cce43ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:16:57.619878 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.619784 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248b1556-db55-4071-8afd-3f350e0d10b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "248b1556-db55-4071-8afd-3f350e0d10b5" (UID: "248b1556-db55-4071-8afd-3f350e0d10b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:16:57.621663 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.621639 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feef996d-af6e-4cf9-9f3c-a1790cce43ed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "feef996d-af6e-4cf9-9f3c-a1790cce43ed" (UID: "feef996d-af6e-4cf9-9f3c-a1790cce43ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 21:16:57.621842 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.621823 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248b1556-db55-4071-8afd-3f350e0d10b5-kube-api-access-dgpkh" (OuterVolumeSpecName: "kube-api-access-dgpkh") pod "248b1556-db55-4071-8afd-3f350e0d10b5" (UID: "248b1556-db55-4071-8afd-3f350e0d10b5"). InnerVolumeSpecName "kube-api-access-dgpkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:16:57.621895 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.621848 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feef996d-af6e-4cf9-9f3c-a1790cce43ed-kube-api-access-hx94n" (OuterVolumeSpecName: "kube-api-access-hx94n") pod "feef996d-af6e-4cf9-9f3c-a1790cce43ed" (UID: "feef996d-af6e-4cf9-9f3c-a1790cce43ed"). InnerVolumeSpecName "kube-api-access-hx94n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:16:57.720769 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.720659 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgpkh\" (UniqueName: \"kubernetes.io/projected/248b1556-db55-4071-8afd-3f350e0d10b5-kube-api-access-dgpkh\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:16:57.720769 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.720702 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feef996d-af6e-4cf9-9f3c-a1790cce43ed-openshift-service-ca\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:16:57.720769 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.720714 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248b1556-db55-4071-8afd-3f350e0d10b5-plugins-config-volume\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:16:57.720769 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.720725 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx94n\" (UniqueName: \"kubernetes.io/projected/feef996d-af6e-4cf9-9f3c-a1790cce43ed-kube-api-access-hx94n\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:16:57.720769 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:57.720739 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feef996d-af6e-4cf9-9f3c-a1790cce43ed-maas-api-tls\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:16:58.489373 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:58.489285 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" Jul 29 21:16:58.489373 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:58.489309 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb" event={"ID":"feef996d-af6e-4cf9-9f3c-a1790cce43ed","Type":"ContainerDied","Data":"97ab33e396efe547f62ac4f7c6e574ffe0759055edd71d5d6e50848f8c70c393"} Jul 29 21:16:58.489911 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:58.489384 2562 scope.go:117] "RemoveContainer" containerID="4f249db4cbadf6dc8146bcbcdcb6f5253d7c30047c2742c7360d3a5a20c3dce6" Jul 29 21:16:58.490911 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:58.490885 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9" event={"ID":"248b1556-db55-4071-8afd-3f350e0d10b5","Type":"ContainerDied","Data":"a4ac6470b6d278895dd2207cf1a61873a7690928cff03076053eba43eeb6a558"} Jul 29 21:16:58.491034 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:58.490948 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9" Jul 29 21:16:58.501279 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:58.501260 2562 scope.go:117] "RemoveContainer" containerID="ad7ab4ad20f6ad1228565a1ca64f788688c59cab5c1a41ae18f34e869479cff3" Jul 29 21:16:58.527005 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:58.526975 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb"] Jul 29 21:16:58.535773 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:58.535743 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23c87980-5dff4dc47-bvwbb"] Jul 29 21:16:58.547640 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:58.547610 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9"] Jul 29 21:16:58.557860 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:58.557828 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-23c87980-6b7954457b-zc6q9"] Jul 29 21:16:59.559102 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:59.559063 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="248b1556-db55-4071-8afd-3f350e0d10b5" path="/var/lib/kubelet/pods/248b1556-db55-4071-8afd-3f350e0d10b5/volumes" Jul 29 21:16:59.559678 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:16:59.559655 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feef996d-af6e-4cf9-9f3c-a1790cce43ed" path="/var/lib/kubelet/pods/feef996d-af6e-4cf9-9f3c-a1790cce43ed/volumes" Jul 29 21:17:01.005659 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.005624 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw"] Jul 29 21:17:01.006104 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.006088 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feef996d-af6e-4cf9-9f3c-a1790cce43ed" containerName="maas-api" Jul 29 21:17:01.006153 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.006105 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="feef996d-af6e-4cf9-9f3c-a1790cce43ed" containerName="maas-api" Jul 29 21:17:01.006153 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.006123 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="248b1556-db55-4071-8afd-3f350e0d10b5" containerName="payload-processing" Jul 29 21:17:01.006153 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.006130 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="248b1556-db55-4071-8afd-3f350e0d10b5" containerName="payload-processing" Jul 29 21:17:01.006249 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.006200 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="feef996d-af6e-4cf9-9f3c-a1790cce43ed" containerName="maas-api" Jul 29 21:17:01.006249 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.006208 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="248b1556-db55-4071-8afd-3f350e0d10b5" containerName="payload-processing" Jul 29 21:17:01.011030 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.011009 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" Jul 29 21:17:01.012901 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.012882 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-946d3187-serving-cert\"" Jul 29 21:17:01.021935 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.021900 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw"] Jul 29 21:17:01.055906 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.055868 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f818496b-670b-4770-881c-0831cd4042fd-openshift-service-ca\") pod \"maas-api-e2e-migrate-946d3187-854cf4bc64-799bw\" (UID: \"f818496b-670b-4770-881c-0831cd4042fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" Jul 29 21:17:01.056102 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.056074 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9fpx\" (UniqueName: \"kubernetes.io/projected/f818496b-670b-4770-881c-0831cd4042fd-kube-api-access-v9fpx\") pod \"maas-api-e2e-migrate-946d3187-854cf4bc64-799bw\" (UID: \"f818496b-670b-4770-881c-0831cd4042fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" Jul 29 21:17:01.056160 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.056116 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f818496b-670b-4770-881c-0831cd4042fd-maas-api-tls\") pod \"maas-api-e2e-migrate-946d3187-854cf4bc64-799bw\" (UID: \"f818496b-670b-4770-881c-0831cd4042fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" Jul 29 21:17:01.157087 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.157053 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f818496b-670b-4770-881c-0831cd4042fd-openshift-service-ca\") pod \"maas-api-e2e-migrate-946d3187-854cf4bc64-799bw\" (UID: \"f818496b-670b-4770-881c-0831cd4042fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" Jul 29 21:17:01.157259 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.157174 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9fpx\" (UniqueName: \"kubernetes.io/projected/f818496b-670b-4770-881c-0831cd4042fd-kube-api-access-v9fpx\") pod \"maas-api-e2e-migrate-946d3187-854cf4bc64-799bw\" (UID: \"f818496b-670b-4770-881c-0831cd4042fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" Jul 29 21:17:01.157259 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.157220 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f818496b-670b-4770-881c-0831cd4042fd-maas-api-tls\") pod \"maas-api-e2e-migrate-946d3187-854cf4bc64-799bw\" (UID: \"f818496b-670b-4770-881c-0831cd4042fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" Jul 29 21:17:01.157362 ip-10-0-137-216 kubenswrapper[2562]: E0729 21:17:01.157345 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-serving-cert: secret "maas-api-e2e-migrate-946d3187-serving-cert" not found Jul 29 21:17:01.157433 ip-10-0-137-216 kubenswrapper[2562]: E0729 21:17:01.157423 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f818496b-670b-4770-881c-0831cd4042fd-maas-api-tls podName:f818496b-670b-4770-881c-0831cd4042fd nodeName:}" failed. No retries permitted until 2026-07-29 21:17:01.657399397 +0000 UTC m=+1838.528502842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f818496b-670b-4770-881c-0831cd4042fd-maas-api-tls") pod "maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" (UID: "f818496b-670b-4770-881c-0831cd4042fd") : secret "maas-api-e2e-migrate-946d3187-serving-cert" not found Jul 29 21:17:01.157877 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.157856 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f818496b-670b-4770-881c-0831cd4042fd-openshift-service-ca\") pod \"maas-api-e2e-migrate-946d3187-854cf4bc64-799bw\" (UID: \"f818496b-670b-4770-881c-0831cd4042fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" Jul 29 21:17:01.167134 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.167108 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9fpx\" (UniqueName: \"kubernetes.io/projected/f818496b-670b-4770-881c-0831cd4042fd-kube-api-access-v9fpx\") pod \"maas-api-e2e-migrate-946d3187-854cf4bc64-799bw\" (UID: \"f818496b-670b-4770-881c-0831cd4042fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" Jul 29 21:17:01.662953 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.662907 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f818496b-670b-4770-881c-0831cd4042fd-maas-api-tls\") pod \"maas-api-e2e-migrate-946d3187-854cf4bc64-799bw\" (UID: \"f818496b-670b-4770-881c-0831cd4042fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" Jul 29 21:17:01.665355 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.665329 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f818496b-670b-4770-881c-0831cd4042fd-maas-api-tls\") pod \"maas-api-e2e-migrate-946d3187-854cf4bc64-799bw\" (UID: \"f818496b-670b-4770-881c-0831cd4042fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" Jul 29 21:17:01.924495 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:01.924400 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" Jul 29 21:17:02.278619 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:02.278579 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw"] Jul 29 21:17:02.279254 ip-10-0-137-216 kubenswrapper[2562]: W0729 21:17:02.279229 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf818496b_670b_4770_881c_0831cd4042fd.slice/crio-9387b58215ef4ade68a84d9e81190909c4cb915e15b68468a879d1b01d7139f5 WatchSource:0}: Error finding container 9387b58215ef4ade68a84d9e81190909c4cb915e15b68468a879d1b01d7139f5: Status 404 returned error can't find the container with id 9387b58215ef4ade68a84d9e81190909c4cb915e15b68468a879d1b01d7139f5 Jul 29 21:17:02.518238 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:02.518203 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" event={"ID":"f818496b-670b-4770-881c-0831cd4042fd","Type":"ContainerStarted","Data":"9387b58215ef4ade68a84d9e81190909c4cb915e15b68468a879d1b01d7139f5"} Jul 29 21:17:03.525629 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:03.525591 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" event={"ID":"f818496b-670b-4770-881c-0831cd4042fd","Type":"ContainerStarted","Data":"6bcf1463b2475862c2952a64bbdb408a76675a572abc63e9648639d3be546e40"} Jul 29 21:17:03.526115 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:03.525772 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" Jul 29 21:17:03.541086 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:03.541016 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" podStartSLOduration=3.065760342 podStartE2EDuration="3.540994938s" totalImagesPullingTime="475.234596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:03.540356794 +0000 UTC m=+1840.411460263" watchObservedRunningTime="2026-07-29 21:17:03.540994938 +0000 UTC m=+1840.412098407" Jul 29 21:17:09.536681 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:09.536641 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" Jul 29 21:17:32.373822 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.373783 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw"] Jul 29 21:17:32.374504 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.374028 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" podUID="f818496b-670b-4770-881c-0831cd4042fd" containerName="maas-api" containerID="cri-o://6bcf1463b2475862c2952a64bbdb408a76675a572abc63e9648639d3be546e40" gracePeriod=30 Jul 29 21:17:32.635725 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.635655 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" Jul 29 21:17:32.669702 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.669665 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9fpx\" (UniqueName: \"kubernetes.io/projected/f818496b-670b-4770-881c-0831cd4042fd-kube-api-access-v9fpx\") pod \"f818496b-670b-4770-881c-0831cd4042fd\" (UID: \"f818496b-670b-4770-881c-0831cd4042fd\") " Jul 29 21:17:32.669907 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.669740 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f818496b-670b-4770-881c-0831cd4042fd-openshift-service-ca\") pod \"f818496b-670b-4770-881c-0831cd4042fd\" (UID: \"f818496b-670b-4770-881c-0831cd4042fd\") " Jul 29 21:17:32.669907 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.669854 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f818496b-670b-4770-881c-0831cd4042fd-maas-api-tls\") pod \"f818496b-670b-4770-881c-0831cd4042fd\" (UID: \"f818496b-670b-4770-881c-0831cd4042fd\") " Jul 29 21:17:32.670138 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.670117 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f818496b-670b-4770-881c-0831cd4042fd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f818496b-670b-4770-881c-0831cd4042fd" (UID: "f818496b-670b-4770-881c-0831cd4042fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:17:32.670293 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.670276 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f818496b-670b-4770-881c-0831cd4042fd-openshift-service-ca\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:17:32.671896 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.671873 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f818496b-670b-4770-881c-0831cd4042fd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f818496b-670b-4770-881c-0831cd4042fd" (UID: "f818496b-670b-4770-881c-0831cd4042fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 21:17:32.672003 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.671963 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f818496b-670b-4770-881c-0831cd4042fd-kube-api-access-v9fpx" (OuterVolumeSpecName: "kube-api-access-v9fpx") pod "f818496b-670b-4770-881c-0831cd4042fd" (UID: "f818496b-670b-4770-881c-0831cd4042fd"). InnerVolumeSpecName "kube-api-access-v9fpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:17:32.715293 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.715255 2562 generic.go:356] "Generic (PLEG): container finished" podID="f818496b-670b-4770-881c-0831cd4042fd" containerID="6bcf1463b2475862c2952a64bbdb408a76675a572abc63e9648639d3be546e40" exitCode=0 Jul 29 21:17:32.715507 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.715306 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" event={"ID":"f818496b-670b-4770-881c-0831cd4042fd","Type":"ContainerDied","Data":"6bcf1463b2475862c2952a64bbdb408a76675a572abc63e9648639d3be546e40"} Jul 29 21:17:32.715507 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.715314 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" Jul 29 21:17:32.715507 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.715382 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw" event={"ID":"f818496b-670b-4770-881c-0831cd4042fd","Type":"ContainerDied","Data":"9387b58215ef4ade68a84d9e81190909c4cb915e15b68468a879d1b01d7139f5"} Jul 29 21:17:32.715507 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.715406 2562 scope.go:117] "RemoveContainer" containerID="6bcf1463b2475862c2952a64bbdb408a76675a572abc63e9648639d3be546e40" Jul 29 21:17:32.729596 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.729572 2562 scope.go:117] "RemoveContainer" containerID="6bcf1463b2475862c2952a64bbdb408a76675a572abc63e9648639d3be546e40" Jul 29 21:17:32.729983 ip-10-0-137-216 kubenswrapper[2562]: E0729 21:17:32.729955 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bcf1463b2475862c2952a64bbdb408a76675a572abc63e9648639d3be546e40\": container with ID starting with 6bcf1463b2475862c2952a64bbdb408a76675a572abc63e9648639d3be546e40 not found: ID does not exist" containerID="6bcf1463b2475862c2952a64bbdb408a76675a572abc63e9648639d3be546e40" Jul 29 21:17:32.730074 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.729994 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bcf1463b2475862c2952a64bbdb408a76675a572abc63e9648639d3be546e40"} err="failed to get container status \"6bcf1463b2475862c2952a64bbdb408a76675a572abc63e9648639d3be546e40\": rpc error: code = NotFound desc = could not find container \"6bcf1463b2475862c2952a64bbdb408a76675a572abc63e9648639d3be546e40\": container with ID starting with 6bcf1463b2475862c2952a64bbdb408a76675a572abc63e9648639d3be546e40 not found: ID does not exist" Jul 29 21:17:32.745526 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.745486 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw"] Jul 29 21:17:32.748863 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.748835 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-946d3187-854cf4bc64-799bw"] Jul 29 21:17:32.771055 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.771024 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9fpx\" (UniqueName: \"kubernetes.io/projected/f818496b-670b-4770-881c-0831cd4042fd-kube-api-access-v9fpx\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:17:32.771055 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:32.771050 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f818496b-670b-4770-881c-0831cd4042fd-maas-api-tls\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:17:33.559339 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:33.559307 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f818496b-670b-4770-881c-0831cd4042fd" path="/var/lib/kubelet/pods/f818496b-670b-4770-881c-0831cd4042fd/volumes" Jul 29 21:17:37.180463 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:37.180429 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq"] Jul 29 21:17:37.180898 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:37.180885 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f818496b-670b-4770-881c-0831cd4042fd" containerName="maas-api" Jul 29 21:17:37.181048 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:37.180898 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f818496b-670b-4770-881c-0831cd4042fd" containerName="maas-api" Jul 29 21:17:37.181048 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:37.180981 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f818496b-670b-4770-881c-0831cd4042fd" containerName="maas-api" Jul 29 21:17:37.185527 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:37.185506 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" Jul 29 21:17:37.187711 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:37.187688 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-17bc2b8b-serving-cert\"" Jul 29 21:17:37.195024 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:37.194999 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq"] Jul 29 21:17:37.314283 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:37.314241 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02117e24-0de7-495a-a7cd-471a9f860150-openshift-service-ca\") pod \"maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq\" (UID: \"02117e24-0de7-495a-a7cd-471a9f860150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" Jul 29 21:17:37.314283 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:37.314282 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmksg\" (UniqueName: \"kubernetes.io/projected/02117e24-0de7-495a-a7cd-471a9f860150-kube-api-access-tmksg\") pod \"maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq\" (UID: \"02117e24-0de7-495a-a7cd-471a9f860150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" Jul 29 21:17:37.314506 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:37.314402 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02117e24-0de7-495a-a7cd-471a9f860150-maas-api-tls\") pod \"maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq\" (UID: \"02117e24-0de7-495a-a7cd-471a9f860150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" Jul 29 21:17:37.415169 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:37.415127 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02117e24-0de7-495a-a7cd-471a9f860150-maas-api-tls\") pod \"maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq\" (UID: \"02117e24-0de7-495a-a7cd-471a9f860150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" Jul 29 21:17:37.415355 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:37.415242 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02117e24-0de7-495a-a7cd-471a9f860150-openshift-service-ca\") pod \"maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq\" (UID: \"02117e24-0de7-495a-a7cd-471a9f860150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" Jul 29 21:17:37.415355 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:37.415265 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmksg\" (UniqueName: \"kubernetes.io/projected/02117e24-0de7-495a-a7cd-471a9f860150-kube-api-access-tmksg\") pod \"maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq\" (UID: \"02117e24-0de7-495a-a7cd-471a9f860150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" Jul 29 21:17:37.416023 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:37.415999 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02117e24-0de7-495a-a7cd-471a9f860150-openshift-service-ca\") pod \"maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq\" (UID: \"02117e24-0de7-495a-a7cd-471a9f860150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" Jul 29 21:17:37.417761 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:37.417738 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02117e24-0de7-495a-a7cd-471a9f860150-maas-api-tls\") pod \"maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq\" (UID: \"02117e24-0de7-495a-a7cd-471a9f860150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" Jul 29 21:17:37.422984 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:37.422963 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmksg\" (UniqueName: \"kubernetes.io/projected/02117e24-0de7-495a-a7cd-471a9f860150-kube-api-access-tmksg\") pod \"maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq\" (UID: \"02117e24-0de7-495a-a7cd-471a9f860150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" Jul 29 21:17:37.499258 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:37.499158 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" Jul 29 21:17:37.643998 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:37.643947 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq"] Jul 29 21:17:37.646292 ip-10-0-137-216 kubenswrapper[2562]: W0729 21:17:37.646255 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02117e24_0de7_495a_a7cd_471a9f860150.slice/crio-f4a4446ea481271dd09569d4b256cbb036ab31a4978eaf1bde4de5cd6fa0fdd1 WatchSource:0}: Error finding container f4a4446ea481271dd09569d4b256cbb036ab31a4978eaf1bde4de5cd6fa0fdd1: Status 404 returned error can't find the container with id f4a4446ea481271dd09569d4b256cbb036ab31a4978eaf1bde4de5cd6fa0fdd1 Jul 29 21:17:37.745521 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:37.745486 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" event={"ID":"02117e24-0de7-495a-a7cd-471a9f860150","Type":"ContainerStarted","Data":"f4a4446ea481271dd09569d4b256cbb036ab31a4978eaf1bde4de5cd6fa0fdd1"} Jul 29 21:17:38.753340 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:38.753301 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" event={"ID":"02117e24-0de7-495a-a7cd-471a9f860150","Type":"ContainerStarted","Data":"6b961252794becb198a26347a86467de7003474b621199facc1b5b8aba842e9c"} Jul 29 21:17:38.753758 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:38.753365 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" Jul 29 21:17:38.771273 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:38.771221 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" podStartSLOduration=1.275962406 podStartE2EDuration="1.771200806s" totalImagesPullingTime="495.2384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:38.77024566 +0000 UTC m=+1875.641349145" watchObservedRunningTime="2026-07-29 21:17:38.771200806 +0000 UTC m=+1875.642304274" Jul 29 21:17:44.765317 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:17:44.765284 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" Jul 29 21:18:07.728111 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:07.728016 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq"] Jul 29 21:18:07.728607 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:07.728269 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" podUID="02117e24-0de7-495a-a7cd-471a9f860150" containerName="maas-api" containerID="cri-o://6b961252794becb198a26347a86467de7003474b621199facc1b5b8aba842e9c" gracePeriod=30 Jul 29 21:18:07.941646 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:07.941568 2562 generic.go:356] "Generic (PLEG): container finished" podID="02117e24-0de7-495a-a7cd-471a9f860150" containerID="6b961252794becb198a26347a86467de7003474b621199facc1b5b8aba842e9c" exitCode=0 Jul 29 21:18:07.941834 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:07.941644 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" event={"ID":"02117e24-0de7-495a-a7cd-471a9f860150","Type":"ContainerDied","Data":"6b961252794becb198a26347a86467de7003474b621199facc1b5b8aba842e9c"} Jul 29 21:18:07.991946 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:07.991853 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" Jul 29 21:18:08.019937 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:08.019890 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmksg\" (UniqueName: \"kubernetes.io/projected/02117e24-0de7-495a-a7cd-471a9f860150-kube-api-access-tmksg\") pod \"02117e24-0de7-495a-a7cd-471a9f860150\" (UID: \"02117e24-0de7-495a-a7cd-471a9f860150\") " Jul 29 21:18:08.020108 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:08.020045 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02117e24-0de7-495a-a7cd-471a9f860150-maas-api-tls\") pod \"02117e24-0de7-495a-a7cd-471a9f860150\" (UID: \"02117e24-0de7-495a-a7cd-471a9f860150\") " Jul 29 21:18:08.020155 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:08.020121 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02117e24-0de7-495a-a7cd-471a9f860150-openshift-service-ca\") pod \"02117e24-0de7-495a-a7cd-471a9f860150\" (UID: \"02117e24-0de7-495a-a7cd-471a9f860150\") " Jul 29 21:18:08.020461 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:08.020433 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02117e24-0de7-495a-a7cd-471a9f860150-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "02117e24-0de7-495a-a7cd-471a9f860150" (UID: "02117e24-0de7-495a-a7cd-471a9f860150"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:18:08.022122 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:08.022095 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02117e24-0de7-495a-a7cd-471a9f860150-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "02117e24-0de7-495a-a7cd-471a9f860150" (UID: "02117e24-0de7-495a-a7cd-471a9f860150"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 21:18:08.022122 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:08.022104 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02117e24-0de7-495a-a7cd-471a9f860150-kube-api-access-tmksg" (OuterVolumeSpecName: "kube-api-access-tmksg") pod "02117e24-0de7-495a-a7cd-471a9f860150" (UID: "02117e24-0de7-495a-a7cd-471a9f860150"). InnerVolumeSpecName "kube-api-access-tmksg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:18:08.121988 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:08.121949 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02117e24-0de7-495a-a7cd-471a9f860150-maas-api-tls\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:18:08.121988 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:08.121981 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02117e24-0de7-495a-a7cd-471a9f860150-openshift-service-ca\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:18:08.121988 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:08.121990 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmksg\" (UniqueName: \"kubernetes.io/projected/02117e24-0de7-495a-a7cd-471a9f860150-kube-api-access-tmksg\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:18:08.948679 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:08.948647 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" Jul 29 21:18:08.949142 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:08.948649 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq" event={"ID":"02117e24-0de7-495a-a7cd-471a9f860150","Type":"ContainerDied","Data":"f4a4446ea481271dd09569d4b256cbb036ab31a4978eaf1bde4de5cd6fa0fdd1"} Jul 29 21:18:08.949142 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:08.948807 2562 scope.go:117] "RemoveContainer" containerID="6b961252794becb198a26347a86467de7003474b621199facc1b5b8aba842e9c" Jul 29 21:18:08.969616 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:08.969584 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq"] Jul 29 21:18:08.972350 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:08.972326 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17bc2b8b-6d9d9856c4-khrgq"] Jul 29 21:18:09.557398 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:09.557356 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02117e24-0de7-495a-a7cd-471a9f860150" path="/var/lib/kubelet/pods/02117e24-0de7-495a-a7cd-471a9f860150/volumes" Jul 29 21:18:13.345399 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:13.345364 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq"] Jul 29 21:18:13.346035 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:13.346014 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02117e24-0de7-495a-a7cd-471a9f860150" containerName="maas-api" Jul 29 21:18:13.346035 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:13.346035 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="02117e24-0de7-495a-a7cd-471a9f860150" containerName="maas-api" Jul 29 21:18:13.346264 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:13.346157 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="02117e24-0de7-495a-a7cd-471a9f860150" containerName="maas-api" Jul 29 21:18:13.351089 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:13.351060 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" Jul 29 21:18:13.352776 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:13.352752 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-2d0e6b56-serving-cert\"" Jul 29 21:18:13.359495 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:13.359464 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq"] Jul 29 21:18:13.472592 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:13.472557 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/307239af-9b85-4ecc-bed0-7933383413df-openshift-service-ca\") pod \"maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq\" (UID: \"307239af-9b85-4ecc-bed0-7933383413df\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" Jul 29 21:18:13.472775 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:13.472612 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blmzn\" (UniqueName: \"kubernetes.io/projected/307239af-9b85-4ecc-bed0-7933383413df-kube-api-access-blmzn\") pod \"maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq\" (UID: \"307239af-9b85-4ecc-bed0-7933383413df\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" Jul 29 21:18:13.472820 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:13.472763 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/307239af-9b85-4ecc-bed0-7933383413df-maas-api-tls\") pod \"maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq\" (UID: \"307239af-9b85-4ecc-bed0-7933383413df\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" Jul 29 21:18:13.574112 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:13.574075 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blmzn\" (UniqueName: \"kubernetes.io/projected/307239af-9b85-4ecc-bed0-7933383413df-kube-api-access-blmzn\") pod \"maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq\" (UID: \"307239af-9b85-4ecc-bed0-7933383413df\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" Jul 29 21:18:13.574326 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:13.574174 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/307239af-9b85-4ecc-bed0-7933383413df-maas-api-tls\") pod \"maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq\" (UID: \"307239af-9b85-4ecc-bed0-7933383413df\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" Jul 29 21:18:13.574326 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:13.574211 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/307239af-9b85-4ecc-bed0-7933383413df-openshift-service-ca\") pod \"maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq\" (UID: \"307239af-9b85-4ecc-bed0-7933383413df\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" Jul 29 21:18:13.574326 ip-10-0-137-216 kubenswrapper[2562]: E0729 21:18:13.574309 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-serving-cert: secret "maas-api-e2e-derive-2d0e6b56-serving-cert" not found Jul 29 21:18:13.574501 ip-10-0-137-216 kubenswrapper[2562]: E0729 21:18:13.574369 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/307239af-9b85-4ecc-bed0-7933383413df-maas-api-tls podName:307239af-9b85-4ecc-bed0-7933383413df nodeName:}" failed. No retries permitted until 2026-07-29 21:18:14.074352719 +0000 UTC m=+1910.945456163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/307239af-9b85-4ecc-bed0-7933383413df-maas-api-tls") pod "maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" (UID: "307239af-9b85-4ecc-bed0-7933383413df") : secret "maas-api-e2e-derive-2d0e6b56-serving-cert" not found Jul 29 21:18:13.574821 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:13.574803 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/307239af-9b85-4ecc-bed0-7933383413df-openshift-service-ca\") pod \"maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq\" (UID: \"307239af-9b85-4ecc-bed0-7933383413df\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" Jul 29 21:18:13.583995 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:13.583968 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blmzn\" (UniqueName: \"kubernetes.io/projected/307239af-9b85-4ecc-bed0-7933383413df-kube-api-access-blmzn\") pod \"maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq\" (UID: \"307239af-9b85-4ecc-bed0-7933383413df\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" Jul 29 21:18:14.080729 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:14.080692 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/307239af-9b85-4ecc-bed0-7933383413df-maas-api-tls\") pod \"maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq\" (UID: \"307239af-9b85-4ecc-bed0-7933383413df\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" Jul 29 21:18:14.083142 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:14.083115 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/307239af-9b85-4ecc-bed0-7933383413df-maas-api-tls\") pod \"maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq\" (UID: \"307239af-9b85-4ecc-bed0-7933383413df\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" Jul 29 21:18:14.266137 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:14.266094 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" Jul 29 21:18:14.413602 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:14.413570 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq"] Jul 29 21:18:14.414396 ip-10-0-137-216 kubenswrapper[2562]: W0729 21:18:14.414365 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307239af_9b85_4ecc_bed0_7933383413df.slice/crio-f7d112f8b3e97ebbcf3d5e15e623bc0537bac0932abcd659e2490fb093175479 WatchSource:0}: Error finding container f7d112f8b3e97ebbcf3d5e15e623bc0537bac0932abcd659e2490fb093175479: Status 404 returned error can't find the container with id f7d112f8b3e97ebbcf3d5e15e623bc0537bac0932abcd659e2490fb093175479 Jul 29 21:18:14.986204 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:14.986167 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" event={"ID":"307239af-9b85-4ecc-bed0-7933383413df","Type":"ContainerStarted","Data":"f7d112f8b3e97ebbcf3d5e15e623bc0537bac0932abcd659e2490fb093175479"} Jul 29 21:18:15.993355 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:15.993317 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" event={"ID":"307239af-9b85-4ecc-bed0-7933383413df","Type":"ContainerStarted","Data":"7d1392af6d9ff91644cb6b76de221ba6a4a9c775d9ba061709e2db50320b112f"} Jul 29 21:18:15.993852 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:15.993391 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" Jul 29 21:18:16.010059 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:16.010005 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" podStartSLOduration=2.095562179 podStartE2EDuration="3.00998876s" totalImagesPullingTime="914.426581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:16.007288661 +0000 UTC m=+1912.878392152" watchObservedRunningTime="2026-07-29 21:18:16.00998876 +0000 UTC m=+1912.881092226" Jul 29 21:18:22.006297 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:22.006262 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" Jul 29 21:18:43.706574 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:43.706422 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq"] Jul 29 21:18:43.707091 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:43.706690 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" podUID="307239af-9b85-4ecc-bed0-7933383413df" containerName="maas-api" containerID="cri-o://7d1392af6d9ff91644cb6b76de221ba6a4a9c775d9ba061709e2db50320b112f" gracePeriod=30 Jul 29 21:18:43.962764 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:43.962691 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" Jul 29 21:18:43.985892 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:43.985860 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/307239af-9b85-4ecc-bed0-7933383413df-openshift-service-ca\") pod \"307239af-9b85-4ecc-bed0-7933383413df\" (UID: \"307239af-9b85-4ecc-bed0-7933383413df\") " Jul 29 21:18:43.986115 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:43.985930 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/307239af-9b85-4ecc-bed0-7933383413df-maas-api-tls\") pod \"307239af-9b85-4ecc-bed0-7933383413df\" (UID: \"307239af-9b85-4ecc-bed0-7933383413df\") " Jul 29 21:18:43.986115 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:43.986033 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blmzn\" (UniqueName: \"kubernetes.io/projected/307239af-9b85-4ecc-bed0-7933383413df-kube-api-access-blmzn\") pod \"307239af-9b85-4ecc-bed0-7933383413df\" (UID: \"307239af-9b85-4ecc-bed0-7933383413df\") " Jul 29 21:18:43.986394 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:43.986271 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/307239af-9b85-4ecc-bed0-7933383413df-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "307239af-9b85-4ecc-bed0-7933383413df" (UID: "307239af-9b85-4ecc-bed0-7933383413df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:18:43.986394 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:43.986381 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/307239af-9b85-4ecc-bed0-7933383413df-openshift-service-ca\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:18:43.988208 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:43.988180 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/307239af-9b85-4ecc-bed0-7933383413df-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "307239af-9b85-4ecc-bed0-7933383413df" (UID: "307239af-9b85-4ecc-bed0-7933383413df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 21:18:43.988307 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:43.988220 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307239af-9b85-4ecc-bed0-7933383413df-kube-api-access-blmzn" (OuterVolumeSpecName: "kube-api-access-blmzn") pod "307239af-9b85-4ecc-bed0-7933383413df" (UID: "307239af-9b85-4ecc-bed0-7933383413df"). InnerVolumeSpecName "kube-api-access-blmzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:18:44.087277 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:44.087244 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blmzn\" (UniqueName: \"kubernetes.io/projected/307239af-9b85-4ecc-bed0-7933383413df-kube-api-access-blmzn\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:18:44.087277 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:44.087276 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/307239af-9b85-4ecc-bed0-7933383413df-maas-api-tls\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:18:44.173223 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:44.173188 2562 generic.go:356] "Generic (PLEG): container finished" podID="307239af-9b85-4ecc-bed0-7933383413df" containerID="7d1392af6d9ff91644cb6b76de221ba6a4a9c775d9ba061709e2db50320b112f" exitCode=0 Jul 29 21:18:44.173411 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:44.173233 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" Jul 29 21:18:44.173411 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:44.173277 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" event={"ID":"307239af-9b85-4ecc-bed0-7933383413df","Type":"ContainerDied","Data":"7d1392af6d9ff91644cb6b76de221ba6a4a9c775d9ba061709e2db50320b112f"} Jul 29 21:18:44.173411 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:44.173325 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq" event={"ID":"307239af-9b85-4ecc-bed0-7933383413df","Type":"ContainerDied","Data":"f7d112f8b3e97ebbcf3d5e15e623bc0537bac0932abcd659e2490fb093175479"} Jul 29 21:18:44.173411 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:44.173352 2562 scope.go:117] "RemoveContainer" containerID="7d1392af6d9ff91644cb6b76de221ba6a4a9c775d9ba061709e2db50320b112f" Jul 29 21:18:44.184406 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:44.184381 2562 scope.go:117] "RemoveContainer" containerID="7d1392af6d9ff91644cb6b76de221ba6a4a9c775d9ba061709e2db50320b112f" Jul 29 21:18:44.184733 ip-10-0-137-216 kubenswrapper[2562]: E0729 21:18:44.184710 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d1392af6d9ff91644cb6b76de221ba6a4a9c775d9ba061709e2db50320b112f\": container with ID starting with 7d1392af6d9ff91644cb6b76de221ba6a4a9c775d9ba061709e2db50320b112f not found: ID does not exist" containerID="7d1392af6d9ff91644cb6b76de221ba6a4a9c775d9ba061709e2db50320b112f" Jul 29 21:18:44.184820 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:44.184742 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d1392af6d9ff91644cb6b76de221ba6a4a9c775d9ba061709e2db50320b112f"} err="failed to get container status \"7d1392af6d9ff91644cb6b76de221ba6a4a9c775d9ba061709e2db50320b112f\": rpc error: code = NotFound desc = could not find container \"7d1392af6d9ff91644cb6b76de221ba6a4a9c775d9ba061709e2db50320b112f\": container with ID starting with 7d1392af6d9ff91644cb6b76de221ba6a4a9c775d9ba061709e2db50320b112f not found: ID does not exist" Jul 29 21:18:44.196131 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:44.196098 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq"] Jul 29 21:18:44.199808 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:44.199775 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2d0e6b56-777449c5c8-k6prq"] Jul 29 21:18:45.559006 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:18:45.558971 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="307239af-9b85-4ecc-bed0-7933383413df" path="/var/lib/kubelet/pods/307239af-9b85-4ecc-bed0-7933383413df/volumes" Jul 29 21:20:40.547207 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:40.547167 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt"] Jul 29 21:20:40.547880 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:40.547858 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="307239af-9b85-4ecc-bed0-7933383413df" containerName="maas-api" Jul 29 21:20:40.548026 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:40.547882 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="307239af-9b85-4ecc-bed0-7933383413df" containerName="maas-api" Jul 29 21:20:40.548088 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:40.548071 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="307239af-9b85-4ecc-bed0-7933383413df" containerName="maas-api" Jul 29 21:20:40.551306 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:40.551276 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" Jul 29 21:20:40.553271 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:40.553252 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f7c67e2c-serving-cert\"" Jul 29 21:20:40.558777 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:40.558751 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt"] Jul 29 21:20:40.642455 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:40.642405 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-openshift-service-ca\") pod \"maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt\" (UID: \"ccf21ece-ee9f-4ec2-ae31-59a3b562199d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" Jul 29 21:20:40.642643 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:40.642465 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glztj\" (UniqueName: \"kubernetes.io/projected/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-kube-api-access-glztj\") pod \"maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt\" (UID: \"ccf21ece-ee9f-4ec2-ae31-59a3b562199d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" Jul 29 21:20:40.643094 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:40.643059 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-maas-api-tls\") pod \"maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt\" (UID: \"ccf21ece-ee9f-4ec2-ae31-59a3b562199d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" Jul 29 21:20:40.744795 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:40.744750 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-maas-api-tls\") pod \"maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt\" (UID: \"ccf21ece-ee9f-4ec2-ae31-59a3b562199d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" Jul 29 21:20:40.745043 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:40.744852 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-openshift-service-ca\") pod \"maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt\" (UID: \"ccf21ece-ee9f-4ec2-ae31-59a3b562199d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" Jul 29 21:20:40.745043 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:40.744878 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glztj\" (UniqueName: \"kubernetes.io/projected/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-kube-api-access-glztj\") pod \"maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt\" (UID: \"ccf21ece-ee9f-4ec2-ae31-59a3b562199d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" Jul 29 21:20:40.745043 ip-10-0-137-216 kubenswrapper[2562]: E0729 21:20:40.744905 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-serving-cert: secret "maas-api-e2e-mt-f7c67e2c-serving-cert" not found Jul 29 21:20:40.745043 ip-10-0-137-216 kubenswrapper[2562]: E0729 21:20:40.745019 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-maas-api-tls podName:ccf21ece-ee9f-4ec2-ae31-59a3b562199d nodeName:}" failed. No retries permitted until 2026-07-29 21:20:41.244998819 +0000 UTC m=+2058.116102266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-maas-api-tls") pod "maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" (UID: "ccf21ece-ee9f-4ec2-ae31-59a3b562199d") : secret "maas-api-e2e-mt-f7c67e2c-serving-cert" not found Jul 29 21:20:40.745631 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:40.745604 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-openshift-service-ca\") pod \"maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt\" (UID: \"ccf21ece-ee9f-4ec2-ae31-59a3b562199d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" Jul 29 21:20:40.760703 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:40.760675 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glztj\" (UniqueName: \"kubernetes.io/projected/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-kube-api-access-glztj\") pod \"maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt\" (UID: \"ccf21ece-ee9f-4ec2-ae31-59a3b562199d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" Jul 29 21:20:41.249455 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:41.249420 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-maas-api-tls\") pod \"maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt\" (UID: \"ccf21ece-ee9f-4ec2-ae31-59a3b562199d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" Jul 29 21:20:41.249643 ip-10-0-137-216 kubenswrapper[2562]: E0729 21:20:41.249578 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-serving-cert: secret "maas-api-e2e-mt-f7c67e2c-serving-cert" not found Jul 29 21:20:41.249692 ip-10-0-137-216 kubenswrapper[2562]: E0729 21:20:41.249652 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-maas-api-tls podName:ccf21ece-ee9f-4ec2-ae31-59a3b562199d nodeName:}" failed. No retries permitted until 2026-07-29 21:20:42.249635787 +0000 UTC m=+2059.120739242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-maas-api-tls") pod "maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" (UID: "ccf21ece-ee9f-4ec2-ae31-59a3b562199d") : secret "maas-api-e2e-mt-f7c67e2c-serving-cert" not found Jul 29 21:20:42.261718 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:42.261674 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-maas-api-tls\") pod \"maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt\" (UID: \"ccf21ece-ee9f-4ec2-ae31-59a3b562199d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" Jul 29 21:20:42.264358 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:42.264330 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-maas-api-tls\") pod \"maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt\" (UID: \"ccf21ece-ee9f-4ec2-ae31-59a3b562199d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" Jul 29 21:20:42.364364 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:42.364322 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" Jul 29 21:20:42.504674 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:42.504641 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt"] Jul 29 21:20:42.506730 ip-10-0-137-216 kubenswrapper[2562]: W0729 21:20:42.506691 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf21ece_ee9f_4ec2_ae31_59a3b562199d.slice/crio-1893ce15dfa5ce5bee7e44fa98edc9505acc6f4d80c9e3ff9fba8f6bf2b0a119 WatchSource:0}: Error finding container 1893ce15dfa5ce5bee7e44fa98edc9505acc6f4d80c9e3ff9fba8f6bf2b0a119: Status 404 returned error can't find the container with id 1893ce15dfa5ce5bee7e44fa98edc9505acc6f4d80c9e3ff9fba8f6bf2b0a119 Jul 29 21:20:42.900561 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:42.900523 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" event={"ID":"ccf21ece-ee9f-4ec2-ae31-59a3b562199d","Type":"ContainerStarted","Data":"1893ce15dfa5ce5bee7e44fa98edc9505acc6f4d80c9e3ff9fba8f6bf2b0a119"} Jul 29 21:20:43.907960 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:43.907888 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" event={"ID":"ccf21ece-ee9f-4ec2-ae31-59a3b562199d","Type":"ContainerStarted","Data":"760ae09b2e731c0aa1fb3b73754d515160b0e612abb20fbd561d018e3e9c47a5"} Jul 29 21:20:43.908388 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:43.907998 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" Jul 29 21:20:43.923760 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:43.923705 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" podStartSLOduration=3.141346506 podStartE2EDuration="3.923686029s" totalImagesPullingTime="782.339523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:20:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:20:43.922838649 +0000 UTC m=+2060.793942135" watchObservedRunningTime="2026-07-29 21:20:43.923686029 +0000 UTC m=+2060.794789496" Jul 29 21:20:49.920755 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:20:49.920725 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" Jul 29 21:21:23.730639 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:21:23.730528 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:21:23.745792 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:21:23.745767 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:26:23.773823 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:26:23.773708 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:26:23.791607 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:26:23.791578 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:30:00.147553 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:30:00.147467 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c67e2c-29756010-sdgpf"] Jul 29 21:30:00.151069 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:30:00.151052 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c67e2c-29756010-sdgpf" Jul 29 21:30:00.152986 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:30:00.152962 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\"" Jul 29 21:30:00.161131 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:30:00.161110 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c67e2c-29756010-sdgpf" Jul 29 21:30:00.163953 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:30:00.163913 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c67e2c-29756010-sdgpf"] Jul 29 21:30:00.302094 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:30:00.302065 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c67e2c-29756010-sdgpf"] Jul 29 21:30:00.303791 ip-10-0-137-216 kubenswrapper[2562]: W0729 21:30:00.303763 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eace496_1f42_40e6_9c75_3735bcbc1bbd.slice/crio-63ec55f6c03775f94f3237947613034d449c4797e08a415ec7e33c13b05e3492 WatchSource:0}: Error finding container 63ec55f6c03775f94f3237947613034d449c4797e08a415ec7e33c13b05e3492: Status 404 returned error can't find the container with id 63ec55f6c03775f94f3237947613034d449c4797e08a415ec7e33c13b05e3492 Jul 29 21:30:00.306294 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:30:00.306274 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 21:30:01.257476 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:30:01.257441 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c67e2c-29756010-sdgpf" event={"ID":"3eace496-1f42-40e6-9c75-3735bcbc1bbd","Type":"ContainerStarted","Data":"63ec55f6c03775f94f3237947613034d449c4797e08a415ec7e33c13b05e3492"} Jul 29 21:30:05.283681 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:30:05.283641 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c67e2c-29756010-sdgpf" event={"ID":"3eace496-1f42-40e6-9c75-3735bcbc1bbd","Type":"ContainerStarted","Data":"b0aee11f12aeda0e1119a31498d510bbeee0c20d19154eac7d5984541984aaf5"} Jul 29 21:30:05.300212 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:30:05.300131 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c67e2c-29756010-sdgpf" podStartSLOduration=0.860699932 podStartE2EDuration="5.300114074s" totalImagesPullingTime="4.439414142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:30:05.297602732 +0000 UTC m=+2622.168706224" watchObservedRunningTime="2026-07-29 21:30:05.300114074 +0000 UTC m=+2622.171217543" Jul 29 21:30:06.290578 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:30:06.290544 2562 generic.go:356] "Generic (PLEG): container finished" podID="3eace496-1f42-40e6-9c75-3735bcbc1bbd" containerID="b0aee11f12aeda0e1119a31498d510bbeee0c20d19154eac7d5984541984aaf5" exitCode=0 Jul 29 21:30:06.291047 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:30:06.290588 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c67e2c-29756010-sdgpf" event={"ID":"3eace496-1f42-40e6-9c75-3735bcbc1bbd","Type":"ContainerDied","Data":"b0aee11f12aeda0e1119a31498d510bbeee0c20d19154eac7d5984541984aaf5"} Jul 29 21:30:07.440311 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:30:07.440284 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c67e2c-29756010-sdgpf" Jul 29 21:30:08.306744 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:30:08.306716 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c67e2c-29756010-sdgpf" Jul 29 21:30:08.306947 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:30:08.306710 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c67e2c-29756010-sdgpf" event={"ID":"3eace496-1f42-40e6-9c75-3735bcbc1bbd","Type":"ContainerDied","Data":"63ec55f6c03775f94f3237947613034d449c4797e08a415ec7e33c13b05e3492"} Jul 29 21:30:08.306947 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:30:08.306838 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63ec55f6c03775f94f3237947613034d449c4797e08a415ec7e33c13b05e3492" Jul 29 21:31:23.816596 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:23.816571 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:31:23.832833 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:23.832808 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:31:38.930571 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:38.930536 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj"] Jul 29 21:31:38.931056 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:38.930773 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj" podUID="b45ed17c-c04e-42af-9ff8-4ae865e3ed50" containerName="manager" containerID="cri-o://5166a21211e941b6ebbc71a48ff77b6f8fe1456931c67bb9ac7f7ece2379d18c" gracePeriod=10 Jul 29 21:31:39.292075 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:39.292048 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj" Jul 29 21:31:39.357435 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:39.357404 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b45ed17c-c04e-42af-9ff8-4ae865e3ed50-extensions-socket-volume\") pod \"b45ed17c-c04e-42af-9ff8-4ae865e3ed50\" (UID: \"b45ed17c-c04e-42af-9ff8-4ae865e3ed50\") " Jul 29 21:31:39.357605 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:39.357459 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75cpt\" (UniqueName: \"kubernetes.io/projected/b45ed17c-c04e-42af-9ff8-4ae865e3ed50-kube-api-access-75cpt\") pod \"b45ed17c-c04e-42af-9ff8-4ae865e3ed50\" (UID: \"b45ed17c-c04e-42af-9ff8-4ae865e3ed50\") " Jul 29 21:31:39.357840 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:39.357813 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45ed17c-c04e-42af-9ff8-4ae865e3ed50-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b45ed17c-c04e-42af-9ff8-4ae865e3ed50" (UID: "b45ed17c-c04e-42af-9ff8-4ae865e3ed50"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:31:39.359641 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:39.359609 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45ed17c-c04e-42af-9ff8-4ae865e3ed50-kube-api-access-75cpt" (OuterVolumeSpecName: "kube-api-access-75cpt") pod "b45ed17c-c04e-42af-9ff8-4ae865e3ed50" (UID: "b45ed17c-c04e-42af-9ff8-4ae865e3ed50"). InnerVolumeSpecName "kube-api-access-75cpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:31:39.458830 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:39.458747 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b45ed17c-c04e-42af-9ff8-4ae865e3ed50-extensions-socket-volume\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:31:39.458830 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:39.458775 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75cpt\" (UniqueName: \"kubernetes.io/projected/b45ed17c-c04e-42af-9ff8-4ae865e3ed50-kube-api-access-75cpt\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:31:39.850687 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:39.850649 2562 generic.go:356] "Generic (PLEG): container finished" podID="b45ed17c-c04e-42af-9ff8-4ae865e3ed50" containerID="5166a21211e941b6ebbc71a48ff77b6f8fe1456931c67bb9ac7f7ece2379d18c" exitCode=0 Jul 29 21:31:39.850687 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:39.850688 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj" Jul 29 21:31:39.850906 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:39.850700 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj" event={"ID":"b45ed17c-c04e-42af-9ff8-4ae865e3ed50","Type":"ContainerDied","Data":"5166a21211e941b6ebbc71a48ff77b6f8fe1456931c67bb9ac7f7ece2379d18c"} Jul 29 21:31:39.850906 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:39.850749 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj" event={"ID":"b45ed17c-c04e-42af-9ff8-4ae865e3ed50","Type":"ContainerDied","Data":"b3b7fbdf2d9ad4d4a68dbfd7e9b9443b76d9b3caaac9710c53122f73de43babc"} Jul 29 21:31:39.850906 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:39.850771 2562 scope.go:117] "RemoveContainer" containerID="5166a21211e941b6ebbc71a48ff77b6f8fe1456931c67bb9ac7f7ece2379d18c" Jul 29 21:31:39.861391 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:39.861379 2562 scope.go:117] "RemoveContainer" containerID="5166a21211e941b6ebbc71a48ff77b6f8fe1456931c67bb9ac7f7ece2379d18c" Jul 29 21:31:39.861658 ip-10-0-137-216 kubenswrapper[2562]: E0729 21:31:39.861633 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5166a21211e941b6ebbc71a48ff77b6f8fe1456931c67bb9ac7f7ece2379d18c\": container with ID starting with 5166a21211e941b6ebbc71a48ff77b6f8fe1456931c67bb9ac7f7ece2379d18c not found: ID does not exist" containerID="5166a21211e941b6ebbc71a48ff77b6f8fe1456931c67bb9ac7f7ece2379d18c" Jul 29 21:31:39.861735 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:39.861668 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5166a21211e941b6ebbc71a48ff77b6f8fe1456931c67bb9ac7f7ece2379d18c"} err="failed to get container status \"5166a21211e941b6ebbc71a48ff77b6f8fe1456931c67bb9ac7f7ece2379d18c\": rpc error: code = NotFound desc = could not find container \"5166a21211e941b6ebbc71a48ff77b6f8fe1456931c67bb9ac7f7ece2379d18c\": container with ID starting with 5166a21211e941b6ebbc71a48ff77b6f8fe1456931c67bb9ac7f7ece2379d18c not found: ID does not exist" Jul 29 21:31:39.870701 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:39.870657 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj"] Jul 29 21:31:39.874177 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:39.874153 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gmcmj"] Jul 29 21:31:41.557825 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:31:41.557780 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b45ed17c-c04e-42af-9ff8-4ae865e3ed50" path="/var/lib/kubelet/pods/b45ed17c-c04e-42af-9ff8-4ae865e3ed50/volumes" Jul 29 21:32:54.556345 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:54.556261 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6c2rk"] Jul 29 21:32:54.556819 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:54.556769 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b45ed17c-c04e-42af-9ff8-4ae865e3ed50" containerName="manager" Jul 29 21:32:54.556819 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:54.556783 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b45ed17c-c04e-42af-9ff8-4ae865e3ed50" containerName="manager" Jul 29 21:32:54.556819 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:54.556793 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eace496-1f42-40e6-9c75-3735bcbc1bbd" containerName="cleanup" Jul 29 21:32:54.556819 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:54.556798 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eace496-1f42-40e6-9c75-3735bcbc1bbd" containerName="cleanup" Jul 29 21:32:54.557010 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:54.556864 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eace496-1f42-40e6-9c75-3735bcbc1bbd" containerName="cleanup" Jul 29 21:32:54.557010 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:54.556875 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b45ed17c-c04e-42af-9ff8-4ae865e3ed50" containerName="manager" Jul 29 21:32:54.560153 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:54.560131 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6c2rk" Jul 29 21:32:54.562334 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:54.562312 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mbgcs\"" Jul 29 21:32:54.571949 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:54.571908 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6c2rk"] Jul 29 21:32:54.703764 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:54.703730 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7jpl\" (UniqueName: \"kubernetes.io/projected/ee4c41be-8d9e-4366-90dd-8d972beeeb21-kube-api-access-p7jpl\") pod \"kuadrant-operator-controller-manager-86878c769f-6c2rk\" (UID: \"ee4c41be-8d9e-4366-90dd-8d972beeeb21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6c2rk" Jul 29 21:32:54.703998 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:54.703963 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee4c41be-8d9e-4366-90dd-8d972beeeb21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6c2rk\" (UID: \"ee4c41be-8d9e-4366-90dd-8d972beeeb21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6c2rk" Jul 29 21:32:54.805269 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:54.805228 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7jpl\" (UniqueName: \"kubernetes.io/projected/ee4c41be-8d9e-4366-90dd-8d972beeeb21-kube-api-access-p7jpl\") pod \"kuadrant-operator-controller-manager-86878c769f-6c2rk\" (UID: \"ee4c41be-8d9e-4366-90dd-8d972beeeb21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6c2rk" Jul 29 21:32:54.805475 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:54.805350 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee4c41be-8d9e-4366-90dd-8d972beeeb21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6c2rk\" (UID: \"ee4c41be-8d9e-4366-90dd-8d972beeeb21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6c2rk" Jul 29 21:32:54.805686 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:54.805667 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee4c41be-8d9e-4366-90dd-8d972beeeb21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6c2rk\" (UID: \"ee4c41be-8d9e-4366-90dd-8d972beeeb21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6c2rk" Jul 29 21:32:54.824663 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:54.824590 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7jpl\" (UniqueName: \"kubernetes.io/projected/ee4c41be-8d9e-4366-90dd-8d972beeeb21-kube-api-access-p7jpl\") pod \"kuadrant-operator-controller-manager-86878c769f-6c2rk\" (UID: \"ee4c41be-8d9e-4366-90dd-8d972beeeb21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6c2rk" Jul 29 21:32:54.870830 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:54.870800 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6c2rk" Jul 29 21:32:55.014811 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:55.014789 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6c2rk"] Jul 29 21:32:55.016876 ip-10-0-137-216 kubenswrapper[2562]: W0729 21:32:55.016851 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4c41be_8d9e_4366_90dd_8d972beeeb21.slice/crio-5ada1c75b324a611d5ad012efbb60dc7664980225d2a099ae5da1c8d6b8ca662 WatchSource:0}: Error finding container 5ada1c75b324a611d5ad012efbb60dc7664980225d2a099ae5da1c8d6b8ca662: Status 404 returned error can't find the container with id 5ada1c75b324a611d5ad012efbb60dc7664980225d2a099ae5da1c8d6b8ca662 Jul 29 21:32:55.324059 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:55.324020 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6c2rk" event={"ID":"ee4c41be-8d9e-4366-90dd-8d972beeeb21","Type":"ContainerStarted","Data":"71fb4e2e642d56e38be91d28b4e00a71bd284b478b6f661640c4abac654a4ffa"} Jul 29 21:32:55.324261 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:55.324085 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6c2rk" event={"ID":"ee4c41be-8d9e-4366-90dd-8d972beeeb21","Type":"ContainerStarted","Data":"5ada1c75b324a611d5ad012efbb60dc7664980225d2a099ae5da1c8d6b8ca662"} Jul 29 21:32:55.324261 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:55.324127 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6c2rk" Jul 29 21:32:55.345218 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:32:55.345110 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6c2rk" podStartSLOduration=1.345093364 podStartE2EDuration="1.345093364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:32:55.344039924 +0000 UTC m=+2792.215143394" watchObservedRunningTime="2026-07-29 21:32:55.345093364 +0000 UTC m=+2792.216196835" Jul 29 21:33:06.332075 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:33:06.332036 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6c2rk" Jul 29 21:35:01.464679 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:01.464642 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt"] Jul 29 21:35:01.465284 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:01.464900 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" podUID="ccf21ece-ee9f-4ec2-ae31-59a3b562199d" containerName="maas-api" containerID="cri-o://760ae09b2e731c0aa1fb3b73754d515160b0e612abb20fbd561d018e3e9c47a5" gracePeriod=30 Jul 29 21:35:01.481301 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:01.481265 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c67e2c-29756010-sdgpf"] Jul 29 21:35:01.488407 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:01.488376 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f7c67e2c-29756010-sdgpf"] Jul 29 21:35:01.561289 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:01.561259 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eace496-1f42-40e6-9c75-3735bcbc1bbd" path="/var/lib/kubelet/pods/3eace496-1f42-40e6-9c75-3735bcbc1bbd/volumes" Jul 29 21:35:01.738859 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:01.738831 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" Jul 29 21:35:01.792585 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:01.792546 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-maas-api-tls\") pod \"ccf21ece-ee9f-4ec2-ae31-59a3b562199d\" (UID: \"ccf21ece-ee9f-4ec2-ae31-59a3b562199d\") " Jul 29 21:35:01.792789 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:01.792602 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-openshift-service-ca\") pod \"ccf21ece-ee9f-4ec2-ae31-59a3b562199d\" (UID: \"ccf21ece-ee9f-4ec2-ae31-59a3b562199d\") " Jul 29 21:35:01.792789 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:01.792713 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glztj\" (UniqueName: \"kubernetes.io/projected/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-kube-api-access-glztj\") pod \"ccf21ece-ee9f-4ec2-ae31-59a3b562199d\" (UID: \"ccf21ece-ee9f-4ec2-ae31-59a3b562199d\") " Jul 29 21:35:01.792959 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:01.792935 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ccf21ece-ee9f-4ec2-ae31-59a3b562199d" (UID: "ccf21ece-ee9f-4ec2-ae31-59a3b562199d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:35:01.793070 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:01.793056 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-openshift-service-ca\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:35:01.794850 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:01.794816 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ccf21ece-ee9f-4ec2-ae31-59a3b562199d" (UID: "ccf21ece-ee9f-4ec2-ae31-59a3b562199d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 21:35:01.794850 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:01.794819 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-kube-api-access-glztj" (OuterVolumeSpecName: "kube-api-access-glztj") pod "ccf21ece-ee9f-4ec2-ae31-59a3b562199d" (UID: "ccf21ece-ee9f-4ec2-ae31-59a3b562199d"). InnerVolumeSpecName "kube-api-access-glztj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:35:01.894003 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:01.893965 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-maas-api-tls\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:35:01.894003 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:01.893997 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glztj\" (UniqueName: \"kubernetes.io/projected/ccf21ece-ee9f-4ec2-ae31-59a3b562199d-kube-api-access-glztj\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:35:02.075884 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:02.075839 2562 generic.go:356] "Generic (PLEG): container finished" podID="ccf21ece-ee9f-4ec2-ae31-59a3b562199d" containerID="760ae09b2e731c0aa1fb3b73754d515160b0e612abb20fbd561d018e3e9c47a5" exitCode=0 Jul 29 21:35:02.075884 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:02.075886 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" Jul 29 21:35:02.076145 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:02.075892 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" event={"ID":"ccf21ece-ee9f-4ec2-ae31-59a3b562199d","Type":"ContainerDied","Data":"760ae09b2e731c0aa1fb3b73754d515160b0e612abb20fbd561d018e3e9c47a5"} Jul 29 21:35:02.076145 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:02.075979 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt" event={"ID":"ccf21ece-ee9f-4ec2-ae31-59a3b562199d","Type":"ContainerDied","Data":"1893ce15dfa5ce5bee7e44fa98edc9505acc6f4d80c9e3ff9fba8f6bf2b0a119"} Jul 29 21:35:02.076145 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:02.076001 2562 scope.go:117] "RemoveContainer" containerID="760ae09b2e731c0aa1fb3b73754d515160b0e612abb20fbd561d018e3e9c47a5" Jul 29 21:35:02.086589 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:02.086569 2562 scope.go:117] "RemoveContainer" containerID="760ae09b2e731c0aa1fb3b73754d515160b0e612abb20fbd561d018e3e9c47a5" Jul 29 21:35:02.086876 ip-10-0-137-216 kubenswrapper[2562]: E0729 21:35:02.086861 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"760ae09b2e731c0aa1fb3b73754d515160b0e612abb20fbd561d018e3e9c47a5\": container with ID starting with 760ae09b2e731c0aa1fb3b73754d515160b0e612abb20fbd561d018e3e9c47a5 not found: ID does not exist" containerID="760ae09b2e731c0aa1fb3b73754d515160b0e612abb20fbd561d018e3e9c47a5" Jul 29 21:35:02.086945 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:02.086883 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"760ae09b2e731c0aa1fb3b73754d515160b0e612abb20fbd561d018e3e9c47a5"} err="failed to get container status \"760ae09b2e731c0aa1fb3b73754d515160b0e612abb20fbd561d018e3e9c47a5\": rpc error: code = NotFound desc = could not find container \"760ae09b2e731c0aa1fb3b73754d515160b0e612abb20fbd561d018e3e9c47a5\": container with ID starting with 760ae09b2e731c0aa1fb3b73754d515160b0e612abb20fbd561d018e3e9c47a5 not found: ID does not exist" Jul 29 21:35:02.102898 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:02.102857 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt"] Jul 29 21:35:02.106668 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:02.106635 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7c67e2c-585875594c-pk5zt"] Jul 29 21:35:03.559147 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:35:03.559117 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccf21ece-ee9f-4ec2-ae31-59a3b562199d" path="/var/lib/kubelet/pods/ccf21ece-ee9f-4ec2-ae31-59a3b562199d/volumes" Jul 29 21:36:23.561044 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:23.561006 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf"] Jul 29 21:36:23.561621 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:23.561602 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccf21ece-ee9f-4ec2-ae31-59a3b562199d" containerName="maas-api" Jul 29 21:36:23.561668 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:23.561624 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccf21ece-ee9f-4ec2-ae31-59a3b562199d" containerName="maas-api" Jul 29 21:36:23.561764 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:23.561753 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccf21ece-ee9f-4ec2-ae31-59a3b562199d" containerName="maas-api" Jul 29 21:36:23.565088 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:23.565065 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf" Jul 29 21:36:23.567170 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:23.567145 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-641c4002\"" Jul 29 21:36:23.567691 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:23.567671 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-641c4002-dockercfg-7rs9r\"" Jul 29 21:36:23.571757 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:23.571736 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf"] Jul 29 21:36:23.592299 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:23.592274 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d4c0c7-6715-41e4-b3d1-97d76e1b111b-plugins-config-volume\") pod \"payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf\" (UID: \"84d4c0c7-6715-41e4-b3d1-97d76e1b111b\") " pod="openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf" Jul 29 21:36:23.592423 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:23.592355 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-285rk\" (UniqueName: \"kubernetes.io/projected/84d4c0c7-6715-41e4-b3d1-97d76e1b111b-kube-api-access-285rk\") pod \"payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf\" (UID: \"84d4c0c7-6715-41e4-b3d1-97d76e1b111b\") " pod="openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf" Jul 29 21:36:23.692983 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:23.692944 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-285rk\" (UniqueName: \"kubernetes.io/projected/84d4c0c7-6715-41e4-b3d1-97d76e1b111b-kube-api-access-285rk\") pod \"payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf\" (UID: \"84d4c0c7-6715-41e4-b3d1-97d76e1b111b\") " pod="openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf" Jul 29 21:36:23.693180 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:23.693065 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d4c0c7-6715-41e4-b3d1-97d76e1b111b-plugins-config-volume\") pod \"payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf\" (UID: \"84d4c0c7-6715-41e4-b3d1-97d76e1b111b\") " pod="openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf" Jul 29 21:36:23.694106 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:23.694080 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d4c0c7-6715-41e4-b3d1-97d76e1b111b-plugins-config-volume\") pod \"payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf\" (UID: \"84d4c0c7-6715-41e4-b3d1-97d76e1b111b\") " pod="openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf" Jul 29 21:36:23.700758 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:23.700731 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-285rk\" (UniqueName: \"kubernetes.io/projected/84d4c0c7-6715-41e4-b3d1-97d76e1b111b-kube-api-access-285rk\") pod \"payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf\" (UID: \"84d4c0c7-6715-41e4-b3d1-97d76e1b111b\") " pod="openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf" Jul 29 21:36:23.856098 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:23.855937 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:36:23.875142 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:23.875116 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:36:23.877841 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:23.877817 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf" Jul 29 21:36:24.067632 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:24.067587 2562 scope.go:117] "RemoveContainer" containerID="b0aee11f12aeda0e1119a31498d510bbeee0c20d19154eac7d5984541984aaf5" Jul 29 21:36:24.229734 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:24.229628 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf"] Jul 29 21:36:24.570292 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:24.570258 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf" event={"ID":"84d4c0c7-6715-41e4-b3d1-97d76e1b111b","Type":"ContainerStarted","Data":"a11324285e5d89ef859dd96c1d8c375a34c9af8d9a98021b2c56540ed93d6729"} Jul 29 21:36:24.570721 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:24.570304 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf" event={"ID":"84d4c0c7-6715-41e4-b3d1-97d76e1b111b","Type":"ContainerStarted","Data":"ec36ddd3a636b42a268fe58a32a3727121eb0cee503ef8eccf0a6d6e9874a77d"} Jul 29 21:36:24.570721 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:24.570387 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf" Jul 29 21:36:24.585401 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:24.585335 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf" podStartSLOduration=1.585315219 podStartE2EDuration="1.585315219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:36:24.584542656 +0000 UTC m=+3001.455646124" watchObservedRunningTime="2026-07-29 21:36:24.585315219 +0000 UTC m=+3001.456418689" Jul 29 21:36:35.578333 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:36:35.578299 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf" Jul 29 21:37:35.061890 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:35.061853 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rbn7l"] Jul 29 21:37:35.065503 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:35.065484 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rbn7l" Jul 29 21:37:35.075252 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:35.075229 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rbn7l"] Jul 29 21:37:35.098366 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:35.098328 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w88tn\" (UniqueName: \"kubernetes.io/projected/aeccb4d8-567d-4bba-b796-5afed7cd9f3d-kube-api-access-w88tn\") pod \"kuadrant-operator-catalog-rbn7l\" (UID: \"aeccb4d8-567d-4bba-b796-5afed7cd9f3d\") " pod="kuadrant-system/kuadrant-operator-catalog-rbn7l" Jul 29 21:37:35.199088 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:35.199043 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w88tn\" (UniqueName: \"kubernetes.io/projected/aeccb4d8-567d-4bba-b796-5afed7cd9f3d-kube-api-access-w88tn\") pod \"kuadrant-operator-catalog-rbn7l\" (UID: \"aeccb4d8-567d-4bba-b796-5afed7cd9f3d\") " pod="kuadrant-system/kuadrant-operator-catalog-rbn7l" Jul 29 21:37:35.210494 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:35.210462 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w88tn\" (UniqueName: \"kubernetes.io/projected/aeccb4d8-567d-4bba-b796-5afed7cd9f3d-kube-api-access-w88tn\") pod \"kuadrant-operator-catalog-rbn7l\" (UID: \"aeccb4d8-567d-4bba-b796-5afed7cd9f3d\") " pod="kuadrant-system/kuadrant-operator-catalog-rbn7l" Jul 29 21:37:35.376584 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:35.376494 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rbn7l" Jul 29 21:37:35.508458 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:35.508434 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rbn7l"] Jul 29 21:37:35.510269 ip-10-0-137-216 kubenswrapper[2562]: W0729 21:37:35.510244 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeccb4d8_567d_4bba_b796_5afed7cd9f3d.slice/crio-86b1a34fe0eac78c1a4fe8176207da73f3894d9fc5ecc9d1b92b68c54e91add2 WatchSource:0}: Error finding container 86b1a34fe0eac78c1a4fe8176207da73f3894d9fc5ecc9d1b92b68c54e91add2: Status 404 returned error can't find the container with id 86b1a34fe0eac78c1a4fe8176207da73f3894d9fc5ecc9d1b92b68c54e91add2 Jul 29 21:37:35.512132 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:35.512113 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 21:37:35.994174 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:35.994143 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rbn7l" event={"ID":"aeccb4d8-567d-4bba-b796-5afed7cd9f3d","Type":"ContainerStarted","Data":"86b1a34fe0eac78c1a4fe8176207da73f3894d9fc5ecc9d1b92b68c54e91add2"} Jul 29 21:37:37.001643 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:37.001602 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rbn7l" event={"ID":"aeccb4d8-567d-4bba-b796-5afed7cd9f3d","Type":"ContainerStarted","Data":"70653f1f779a18d76c33a490d954b6be2a588ca107ec8d04a665a687cb2ea7a9"} Jul 29 21:37:37.019048 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:37.018998 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-rbn7l" podStartSLOduration=1.4955648209999999 podStartE2EDuration="2.01898232s" totalImagesPullingTime="523.417499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:37:37.01656782 +0000 UTC m=+3073.887671289" watchObservedRunningTime="2026-07-29 21:37:37.01898232 +0000 UTC m=+3073.890085786" Jul 29 21:37:45.377104 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:45.377065 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-rbn7l" Jul 29 21:37:45.377688 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:45.377135 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-rbn7l" Jul 29 21:37:45.399611 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:45.399578 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-rbn7l" Jul 29 21:37:46.075553 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:46.075519 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-rbn7l" Jul 29 21:37:46.108947 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:46.108891 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rbn7l"] Jul 29 21:37:48.066531 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:48.066495 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-rbn7l" podUID="aeccb4d8-567d-4bba-b796-5afed7cd9f3d" containerName="registry-server" containerID="cri-o://70653f1f779a18d76c33a490d954b6be2a588ca107ec8d04a665a687cb2ea7a9" gracePeriod=2 Jul 29 21:37:48.334080 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:48.334048 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rbn7l" Jul 29 21:37:48.437884 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:48.437850 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w88tn\" (UniqueName: \"kubernetes.io/projected/aeccb4d8-567d-4bba-b796-5afed7cd9f3d-kube-api-access-w88tn\") pod \"aeccb4d8-567d-4bba-b796-5afed7cd9f3d\" (UID: \"aeccb4d8-567d-4bba-b796-5afed7cd9f3d\") " Jul 29 21:37:48.439943 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:48.439905 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeccb4d8-567d-4bba-b796-5afed7cd9f3d-kube-api-access-w88tn" (OuterVolumeSpecName: "kube-api-access-w88tn") pod "aeccb4d8-567d-4bba-b796-5afed7cd9f3d" (UID: "aeccb4d8-567d-4bba-b796-5afed7cd9f3d"). InnerVolumeSpecName "kube-api-access-w88tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:37:48.539302 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:48.539263 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w88tn\" (UniqueName: \"kubernetes.io/projected/aeccb4d8-567d-4bba-b796-5afed7cd9f3d-kube-api-access-w88tn\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:37:49.073222 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:49.073190 2562 generic.go:356] "Generic (PLEG): container finished" podID="aeccb4d8-567d-4bba-b796-5afed7cd9f3d" containerID="70653f1f779a18d76c33a490d954b6be2a588ca107ec8d04a665a687cb2ea7a9" exitCode=0 Jul 29 21:37:49.073628 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:49.073229 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rbn7l" event={"ID":"aeccb4d8-567d-4bba-b796-5afed7cd9f3d","Type":"ContainerDied","Data":"70653f1f779a18d76c33a490d954b6be2a588ca107ec8d04a665a687cb2ea7a9"} Jul 29 21:37:49.073628 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:49.073241 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rbn7l" Jul 29 21:37:49.073628 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:49.073287 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rbn7l" event={"ID":"aeccb4d8-567d-4bba-b796-5afed7cd9f3d","Type":"ContainerDied","Data":"86b1a34fe0eac78c1a4fe8176207da73f3894d9fc5ecc9d1b92b68c54e91add2"} Jul 29 21:37:49.073628 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:49.073311 2562 scope.go:117] "RemoveContainer" containerID="70653f1f779a18d76c33a490d954b6be2a588ca107ec8d04a665a687cb2ea7a9" Jul 29 21:37:49.083293 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:49.083270 2562 scope.go:117] "RemoveContainer" containerID="70653f1f779a18d76c33a490d954b6be2a588ca107ec8d04a665a687cb2ea7a9" Jul 29 21:37:49.083545 ip-10-0-137-216 kubenswrapper[2562]: E0729 21:37:49.083525 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70653f1f779a18d76c33a490d954b6be2a588ca107ec8d04a665a687cb2ea7a9\": container with ID starting with 70653f1f779a18d76c33a490d954b6be2a588ca107ec8d04a665a687cb2ea7a9 not found: ID does not exist" containerID="70653f1f779a18d76c33a490d954b6be2a588ca107ec8d04a665a687cb2ea7a9" Jul 29 21:37:49.083612 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:49.083554 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70653f1f779a18d76c33a490d954b6be2a588ca107ec8d04a665a687cb2ea7a9"} err="failed to get container status \"70653f1f779a18d76c33a490d954b6be2a588ca107ec8d04a665a687cb2ea7a9\": rpc error: code = NotFound desc = could not find container \"70653f1f779a18d76c33a490d954b6be2a588ca107ec8d04a665a687cb2ea7a9\": container with ID starting with 70653f1f779a18d76c33a490d954b6be2a588ca107ec8d04a665a687cb2ea7a9 not found: ID does not exist" Jul 29 21:37:49.094473 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:49.094454 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rbn7l"] Jul 29 21:37:49.098882 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:49.098862 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rbn7l"] Jul 29 21:37:49.557422 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:37:49.557384 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeccb4d8-567d-4bba-b796-5afed7cd9f3d" path="/var/lib/kubelet/pods/aeccb4d8-567d-4bba-b796-5afed7cd9f3d/volumes" Jul 29 21:41:23.895911 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:41:23.895797 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:41:23.917149 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:41:23.917126 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:45:00.141120 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:00.141029 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bdcc5a3e-29756025-wd5zc"] Jul 29 21:45:00.141812 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:00.141653 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeccb4d8-567d-4bba-b796-5afed7cd9f3d" containerName="registry-server" Jul 29 21:45:00.141812 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:00.141675 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeccb4d8-567d-4bba-b796-5afed7cd9f3d" containerName="registry-server" Jul 29 21:45:00.141911 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:00.141817 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeccb4d8-567d-4bba-b796-5afed7cd9f3d" containerName="registry-server" Jul 29 21:45:00.145211 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:00.145192 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bdcc5a3e-29756025-wd5zc" Jul 29 21:45:00.147594 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:00.147573 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\"" Jul 29 21:45:00.155934 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:00.155902 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bdcc5a3e-29756025-wd5zc" Jul 29 21:45:00.161152 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:00.161122 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bdcc5a3e-29756025-wd5zc"] Jul 29 21:45:00.306163 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:00.299770 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bdcc5a3e-29756025-wd5zc"] Jul 29 21:45:00.309694 ip-10-0-137-216 kubenswrapper[2562]: W0729 21:45:00.309635 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3393b01c_62ba_4b07_86bc_5d756a289a23.slice/crio-ec832b22eda719e625f07ec1ce401a30f60922bc47f88f022bd32718ca2744f0 WatchSource:0}: Error finding container ec832b22eda719e625f07ec1ce401a30f60922bc47f88f022bd32718ca2744f0: Status 404 returned error can't find the container with id ec832b22eda719e625f07ec1ce401a30f60922bc47f88f022bd32718ca2744f0 Jul 29 21:45:00.627283 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:00.627243 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bdcc5a3e-29756025-wd5zc" event={"ID":"3393b01c-62ba-4b07-86bc-5d756a289a23","Type":"ContainerStarted","Data":"aa48d0e0dd3f10e03d4daa29eccc760416d00c6228a4854ccbdb9bc3d732c6b6"} Jul 29 21:45:00.627484 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:00.627302 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bdcc5a3e-29756025-wd5zc" event={"ID":"3393b01c-62ba-4b07-86bc-5d756a289a23","Type":"ContainerStarted","Data":"ec832b22eda719e625f07ec1ce401a30f60922bc47f88f022bd32718ca2744f0"} Jul 29 21:45:00.642065 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:00.642015 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bdcc5a3e-29756025-wd5zc" podStartSLOduration=0.64199997 podStartE2EDuration="641.99997ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:45:00.640177266 +0000 UTC m=+3517.511280734" watchObservedRunningTime="2026-07-29 21:45:00.64199997 +0000 UTC m=+3517.513103436" Jul 29 21:45:01.634374 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:01.634340 2562 generic.go:356] "Generic (PLEG): container finished" podID="3393b01c-62ba-4b07-86bc-5d756a289a23" containerID="aa48d0e0dd3f10e03d4daa29eccc760416d00c6228a4854ccbdb9bc3d732c6b6" exitCode=0 Jul 29 21:45:01.634792 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:01.634374 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bdcc5a3e-29756025-wd5zc" event={"ID":"3393b01c-62ba-4b07-86bc-5d756a289a23","Type":"ContainerDied","Data":"aa48d0e0dd3f10e03d4daa29eccc760416d00c6228a4854ccbdb9bc3d732c6b6"} Jul 29 21:45:02.794788 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:02.794764 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bdcc5a3e-29756025-wd5zc" Jul 29 21:45:02.991672 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:02.991570 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-641c4002-9wx54"] Jul 29 21:45:02.992107 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:02.992088 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3393b01c-62ba-4b07-86bc-5d756a289a23" containerName="cleanup" Jul 29 21:45:02.992107 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:02.992107 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3393b01c-62ba-4b07-86bc-5d756a289a23" containerName="cleanup" Jul 29 21:45:02.992265 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:02.992191 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3393b01c-62ba-4b07-86bc-5d756a289a23" containerName="cleanup" Jul 29 21:45:02.995174 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:02.995158 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-641c4002-9wx54" Jul 29 21:45:03.002661 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:03.002629 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-641c4002-9wx54"] Jul 29 21:45:03.092994 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:03.092949 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbda271d-e80c-408a-8146-c0c08a3b2a0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-641c4002-9wx54\" (UID: \"cbda271d-e80c-408a-8146-c0c08a3b2a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-641c4002-9wx54" Jul 29 21:45:03.194104 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:03.194004 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbda271d-e80c-408a-8146-c0c08a3b2a0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-641c4002-9wx54\" (UID: \"cbda271d-e80c-408a-8146-c0c08a3b2a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-641c4002-9wx54" Jul 29 21:45:03.194652 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:03.194630 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbda271d-e80c-408a-8146-c0c08a3b2a0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-641c4002-9wx54\" (UID: \"cbda271d-e80c-408a-8146-c0c08a3b2a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-641c4002-9wx54" Jul 29 21:45:03.310669 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:03.310640 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-641c4002-9wx54" Jul 29 21:45:03.456848 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:03.456815 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-641c4002-9wx54"] Jul 29 21:45:03.458164 ip-10-0-137-216 kubenswrapper[2562]: W0729 21:45:03.458136 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbda271d_e80c_408a_8146_c0c08a3b2a0e.slice/crio-877f412d0b4ae3ab99583db29e139b86781d77fb02bddf0f06907140d386692a WatchSource:0}: Error finding container 877f412d0b4ae3ab99583db29e139b86781d77fb02bddf0f06907140d386692a: Status 404 returned error can't find the container with id 877f412d0b4ae3ab99583db29e139b86781d77fb02bddf0f06907140d386692a Jul 29 21:45:03.647588 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:03.647560 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bdcc5a3e-29756025-wd5zc" Jul 29 21:45:03.647772 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:03.647560 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bdcc5a3e-29756025-wd5zc" event={"ID":"3393b01c-62ba-4b07-86bc-5d756a289a23","Type":"ContainerDied","Data":"ec832b22eda719e625f07ec1ce401a30f60922bc47f88f022bd32718ca2744f0"} Jul 29 21:45:03.647772 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:03.647702 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec832b22eda719e625f07ec1ce401a30f60922bc47f88f022bd32718ca2744f0" Jul 29 21:45:03.648975 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:03.648949 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-641c4002-9wx54" event={"ID":"cbda271d-e80c-408a-8146-c0c08a3b2a0e","Type":"ContainerStarted","Data":"877f412d0b4ae3ab99583db29e139b86781d77fb02bddf0f06907140d386692a"} Jul 29 21:45:04.657319 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:04.657280 2562 generic.go:356] "Generic (PLEG): container finished" podID="cbda271d-e80c-408a-8146-c0c08a3b2a0e" containerID="ee580e5ebcaccde240520a19d82f90c8a13808373c41eb50ee3a3d8fe5c2e57a" exitCode=0 Jul 29 21:45:04.657749 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:04.657368 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-641c4002-9wx54" event={"ID":"cbda271d-e80c-408a-8146-c0c08a3b2a0e","Type":"ContainerDied","Data":"ee580e5ebcaccde240520a19d82f90c8a13808373c41eb50ee3a3d8fe5c2e57a"} Jul 29 21:45:05.806523 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:05.806498 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-641c4002-9wx54" Jul 29 21:45:05.821425 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:05.821393 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbda271d-e80c-408a-8146-c0c08a3b2a0e-maas-api-service-ca\") pod \"cbda271d-e80c-408a-8146-c0c08a3b2a0e\" (UID: \"cbda271d-e80c-408a-8146-c0c08a3b2a0e\") " Jul 29 21:45:05.821867 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:05.821831 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbda271d-e80c-408a-8146-c0c08a3b2a0e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cbda271d-e80c-408a-8146-c0c08a3b2a0e" (UID: "cbda271d-e80c-408a-8146-c0c08a3b2a0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:45:05.922496 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:05.922405 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbda271d-e80c-408a-8146-c0c08a3b2a0e-maas-api-service-ca\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:45:06.670788 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:06.670750 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-641c4002-9wx54" event={"ID":"cbda271d-e80c-408a-8146-c0c08a3b2a0e","Type":"ContainerDied","Data":"877f412d0b4ae3ab99583db29e139b86781d77fb02bddf0f06907140d386692a"} Jul 29 21:45:06.670788 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:06.670778 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-641c4002-9wx54" Jul 29 21:45:06.671040 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:06.670810 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="877f412d0b4ae3ab99583db29e139b86781d77fb02bddf0f06907140d386692a" Jul 29 21:45:18.521354 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:18.521321 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf"] Jul 29 21:45:18.521830 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:18.521567 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf" podUID="84d4c0c7-6715-41e4-b3d1-97d76e1b111b" containerName="payload-processing" containerID="cri-o://a11324285e5d89ef859dd96c1d8c375a34c9af8d9a98021b2c56540ed93d6729" gracePeriod=30 Jul 29 21:45:18.694587 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:18.694525 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-641c4002-9wx54"] Jul 29 21:45:18.699371 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:18.699331 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-641c4002-9wx54"] Jul 29 21:45:18.750403 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:18.750372 2562 generic.go:356] "Generic (PLEG): container finished" podID="84d4c0c7-6715-41e4-b3d1-97d76e1b111b" containerID="a11324285e5d89ef859dd96c1d8c375a34c9af8d9a98021b2c56540ed93d6729" exitCode=0 Jul 29 21:45:18.750579 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:18.750446 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf" event={"ID":"84d4c0c7-6715-41e4-b3d1-97d76e1b111b","Type":"ContainerDied","Data":"a11324285e5d89ef859dd96c1d8c375a34c9af8d9a98021b2c56540ed93d6729"} Jul 29 21:45:18.787075 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:18.787014 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf" Jul 29 21:45:18.849684 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:18.849653 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-285rk\" (UniqueName: \"kubernetes.io/projected/84d4c0c7-6715-41e4-b3d1-97d76e1b111b-kube-api-access-285rk\") pod \"84d4c0c7-6715-41e4-b3d1-97d76e1b111b\" (UID: \"84d4c0c7-6715-41e4-b3d1-97d76e1b111b\") " Jul 29 21:45:18.849844 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:18.849757 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d4c0c7-6715-41e4-b3d1-97d76e1b111b-plugins-config-volume\") pod \"84d4c0c7-6715-41e4-b3d1-97d76e1b111b\" (UID: \"84d4c0c7-6715-41e4-b3d1-97d76e1b111b\") " Jul 29 21:45:18.850158 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:18.850133 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d4c0c7-6715-41e4-b3d1-97d76e1b111b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84d4c0c7-6715-41e4-b3d1-97d76e1b111b" (UID: "84d4c0c7-6715-41e4-b3d1-97d76e1b111b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:45:18.851758 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:18.851738 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d4c0c7-6715-41e4-b3d1-97d76e1b111b-kube-api-access-285rk" (OuterVolumeSpecName: "kube-api-access-285rk") pod "84d4c0c7-6715-41e4-b3d1-97d76e1b111b" (UID: "84d4c0c7-6715-41e4-b3d1-97d76e1b111b"). InnerVolumeSpecName "kube-api-access-285rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:45:18.950593 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:18.950554 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d4c0c7-6715-41e4-b3d1-97d76e1b111b-plugins-config-volume\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:45:18.950593 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:18.950586 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-285rk\" (UniqueName: \"kubernetes.io/projected/84d4c0c7-6715-41e4-b3d1-97d76e1b111b-kube-api-access-285rk\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:45:19.557344 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:19.557307 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbda271d-e80c-408a-8146-c0c08a3b2a0e" path="/var/lib/kubelet/pods/cbda271d-e80c-408a-8146-c0c08a3b2a0e/volumes" Jul 29 21:45:19.757341 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:19.757311 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf" Jul 29 21:45:19.757515 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:19.757348 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf" event={"ID":"84d4c0c7-6715-41e4-b3d1-97d76e1b111b","Type":"ContainerDied","Data":"ec36ddd3a636b42a268fe58a32a3727121eb0cee503ef8eccf0a6d6e9874a77d"} Jul 29 21:45:19.757515 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:19.757413 2562 scope.go:117] "RemoveContainer" containerID="a11324285e5d89ef859dd96c1d8c375a34c9af8d9a98021b2c56540ed93d6729" Jul 29 21:45:19.774610 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:19.774579 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf"] Jul 29 21:45:19.777931 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:19.777896 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-641c4002-6c4bf6b8bb-khgxf"] Jul 29 21:45:21.564778 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:45:21.564740 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84d4c0c7-6715-41e4-b3d1-97d76e1b111b" path="/var/lib/kubelet/pods/84d4c0c7-6715-41e4-b3d1-97d76e1b111b/volumes" Jul 29 21:46:01.101202 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:01.101166 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bdcc5a3e-29756025-wd5zc"] Jul 29 21:46:01.112249 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:01.112219 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-bdcc5a3e-29756025-wd5zc"] Jul 29 21:46:01.565117 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:01.565073 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3393b01c-62ba-4b07-86bc-5d756a289a23" path="/var/lib/kubelet/pods/3393b01c-62ba-4b07-86bc-5d756a289a23/volumes" Jul 29 21:46:23.935456 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:23.935348 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:46:23.959605 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:23.959583 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:46:37.229789 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:37.229741 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld"] Jul 29 21:46:37.230450 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:37.230430 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbda271d-e80c-408a-8146-c0c08a3b2a0e" containerName="revoke-keys" Jul 29 21:46:37.230450 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:37.230451 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbda271d-e80c-408a-8146-c0c08a3b2a0e" containerName="revoke-keys" Jul 29 21:46:37.230596 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:37.230470 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84d4c0c7-6715-41e4-b3d1-97d76e1b111b" containerName="payload-processing" Jul 29 21:46:37.230596 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:37.230480 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="84d4c0c7-6715-41e4-b3d1-97d76e1b111b" containerName="payload-processing" Jul 29 21:46:37.230596 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:37.230592 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbda271d-e80c-408a-8146-c0c08a3b2a0e" containerName="revoke-keys" Jul 29 21:46:37.230760 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:37.230608 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="84d4c0c7-6715-41e4-b3d1-97d76e1b111b" containerName="payload-processing" Jul 29 21:46:37.234021 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:37.233999 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld" Jul 29 21:46:37.236168 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:37.236132 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ceb7ca\"" Jul 29 21:46:37.236305 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:37.236169 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ceb7ca-dockercfg-cvjh6\"" Jul 29 21:46:37.243898 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:37.243861 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld"] Jul 29 21:46:37.294262 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:37.294226 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b35c2e71-0617-4eb3-bc90-3c1423edcdcb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld\" (UID: \"b35c2e71-0617-4eb3-bc90-3c1423edcdcb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld" Jul 29 21:46:37.294262 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:37.294265 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fwmv\" (UniqueName: \"kubernetes.io/projected/b35c2e71-0617-4eb3-bc90-3c1423edcdcb-kube-api-access-2fwmv\") pod \"payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld\" (UID: \"b35c2e71-0617-4eb3-bc90-3c1423edcdcb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld" Jul 29 21:46:37.395017 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:37.394979 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b35c2e71-0617-4eb3-bc90-3c1423edcdcb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld\" (UID: \"b35c2e71-0617-4eb3-bc90-3c1423edcdcb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld" Jul 29 21:46:37.395228 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:37.395028 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fwmv\" (UniqueName: \"kubernetes.io/projected/b35c2e71-0617-4eb3-bc90-3c1423edcdcb-kube-api-access-2fwmv\") pod \"payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld\" (UID: \"b35c2e71-0617-4eb3-bc90-3c1423edcdcb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld" Jul 29 21:46:37.395709 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:37.395690 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b35c2e71-0617-4eb3-bc90-3c1423edcdcb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld\" (UID: \"b35c2e71-0617-4eb3-bc90-3c1423edcdcb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld" Jul 29 21:46:37.402955 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:37.402892 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fwmv\" (UniqueName: \"kubernetes.io/projected/b35c2e71-0617-4eb3-bc90-3c1423edcdcb-kube-api-access-2fwmv\") pod \"payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld\" (UID: \"b35c2e71-0617-4eb3-bc90-3c1423edcdcb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld" Jul 29 21:46:37.547151 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:37.547066 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld" Jul 29 21:46:37.694975 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:37.694944 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld"] Jul 29 21:46:37.696971 ip-10-0-137-216 kubenswrapper[2562]: W0729 21:46:37.696940 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35c2e71_0617_4eb3_bc90_3c1423edcdcb.slice/crio-8bdcc56f14da94b71e1ebaa5001be93cbc467f644e6926fb7d260a51c51b2a63 WatchSource:0}: Error finding container 8bdcc56f14da94b71e1ebaa5001be93cbc467f644e6926fb7d260a51c51b2a63: Status 404 returned error can't find the container with id 8bdcc56f14da94b71e1ebaa5001be93cbc467f644e6926fb7d260a51c51b2a63 Jul 29 21:46:38.242784 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:38.242751 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld" event={"ID":"b35c2e71-0617-4eb3-bc90-3c1423edcdcb","Type":"ContainerStarted","Data":"c50376ed92779193456c62b83bb3125b7da812964d9403840bd371e82ea737af"} Jul 29 21:46:38.243207 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:38.242804 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld" event={"ID":"b35c2e71-0617-4eb3-bc90-3c1423edcdcb","Type":"ContainerStarted","Data":"8bdcc56f14da94b71e1ebaa5001be93cbc467f644e6926fb7d260a51c51b2a63"} Jul 29 21:46:38.243207 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:38.242835 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld" Jul 29 21:46:38.259536 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:38.259469 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld" podStartSLOduration=1.259452786 podStartE2EDuration="1.259452786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:46:38.256863539 +0000 UTC m=+3615.127967042" watchObservedRunningTime="2026-07-29 21:46:38.259452786 +0000 UTC m=+3615.130556253" Jul 29 21:46:49.250452 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:46:49.250418 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld" Jul 29 21:48:16.134093 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:16.134058 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cb4c-6rt4j"] Jul 29 21:48:16.137585 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:16.137567 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cb4c-6rt4j" Jul 29 21:48:16.139628 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:16.139595 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\"" Jul 29 21:48:16.143756 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:16.143730 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cb4c-6rt4j"] Jul 29 21:48:16.186120 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:16.186083 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b44a832-f1db-4338-8b39-0025412be068-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0cb4c-6rt4j\" (UID: \"1b44a832-f1db-4338-8b39-0025412be068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cb4c-6rt4j" Jul 29 21:48:16.287445 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:16.287413 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b44a832-f1db-4338-8b39-0025412be068-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0cb4c-6rt4j\" (UID: \"1b44a832-f1db-4338-8b39-0025412be068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cb4c-6rt4j" Jul 29 21:48:16.288118 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:16.288095 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b44a832-f1db-4338-8b39-0025412be068-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0cb4c-6rt4j\" (UID: \"1b44a832-f1db-4338-8b39-0025412be068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cb4c-6rt4j" Jul 29 21:48:16.450557 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:16.450459 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cb4c-6rt4j" Jul 29 21:48:16.798241 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:16.798213 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cb4c-6rt4j"] Jul 29 21:48:16.799448 ip-10-0-137-216 kubenswrapper[2562]: W0729 21:48:16.799423 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b44a832_f1db_4338_8b39_0025412be068.slice/crio-82a7c4b7a745f63b43f838fa9a768f9a983592d6554c2c26c4027d9fac0bec17 WatchSource:0}: Error finding container 82a7c4b7a745f63b43f838fa9a768f9a983592d6554c2c26c4027d9fac0bec17: Status 404 returned error can't find the container with id 82a7c4b7a745f63b43f838fa9a768f9a983592d6554c2c26c4027d9fac0bec17 Jul 29 21:48:16.831747 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:16.831723 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cb4c-6rt4j" event={"ID":"1b44a832-f1db-4338-8b39-0025412be068","Type":"ContainerStarted","Data":"82a7c4b7a745f63b43f838fa9a768f9a983592d6554c2c26c4027d9fac0bec17"} Jul 29 21:48:17.839567 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:17.839533 2562 generic.go:356] "Generic (PLEG): container finished" podID="1b44a832-f1db-4338-8b39-0025412be068" containerID="e97d1b6314e3b0c93789bdf9c9c74e48f7757a2d34968e1711451ea635bc9eaa" exitCode=0 Jul 29 21:48:17.840006 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:17.839628 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cb4c-6rt4j" event={"ID":"1b44a832-f1db-4338-8b39-0025412be068","Type":"ContainerDied","Data":"e97d1b6314e3b0c93789bdf9c9c74e48f7757a2d34968e1711451ea635bc9eaa"} Jul 29 21:48:18.988664 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:18.988635 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cb4c-6rt4j" Jul 29 21:48:19.016539 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:19.016508 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b44a832-f1db-4338-8b39-0025412be068-maas-api-service-ca\") pod \"1b44a832-f1db-4338-8b39-0025412be068\" (UID: \"1b44a832-f1db-4338-8b39-0025412be068\") " Jul 29 21:48:19.016859 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:19.016834 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b44a832-f1db-4338-8b39-0025412be068-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b44a832-f1db-4338-8b39-0025412be068" (UID: "1b44a832-f1db-4338-8b39-0025412be068"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:48:19.117489 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:19.117402 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b44a832-f1db-4338-8b39-0025412be068-maas-api-service-ca\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:48:19.852333 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:19.852243 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cb4c-6rt4j" Jul 29 21:48:19.852333 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:19.852295 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cb4c-6rt4j" event={"ID":"1b44a832-f1db-4338-8b39-0025412be068","Type":"ContainerDied","Data":"82a7c4b7a745f63b43f838fa9a768f9a983592d6554c2c26c4027d9fac0bec17"} Jul 29 21:48:19.852541 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:19.852343 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82a7c4b7a745f63b43f838fa9a768f9a983592d6554c2c26c4027d9fac0bec17" Jul 29 21:48:26.825435 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:26.825401 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cb4c-6rt4j"] Jul 29 21:48:26.832434 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:26.832412 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cb4c-6rt4j"] Jul 29 21:48:27.558192 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:27.558158 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b44a832-f1db-4338-8b39-0025412be068" path="/var/lib/kubelet/pods/1b44a832-f1db-4338-8b39-0025412be068/volumes" Jul 29 21:48:33.611892 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:33.611842 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ceb7ca-9l7n8"] Jul 29 21:48:33.612345 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:33.612329 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b44a832-f1db-4338-8b39-0025412be068" containerName="revoke-keys" Jul 29 21:48:33.612394 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:33.612347 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b44a832-f1db-4338-8b39-0025412be068" containerName="revoke-keys" Jul 29 21:48:33.612436 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:33.612428 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b44a832-f1db-4338-8b39-0025412be068" containerName="revoke-keys" Jul 29 21:48:33.616836 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:33.616813 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ceb7ca-9l7n8" Jul 29 21:48:33.618982 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:33.618949 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\"" Jul 29 21:48:33.622892 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:33.622865 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ceb7ca-9l7n8"] Jul 29 21:48:33.766381 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:33.766343 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6abb428f-5ee7-4186-b81f-d5f0dc03bcb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ceb7ca-9l7n8\" (UID: \"6abb428f-5ee7-4186-b81f-d5f0dc03bcb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ceb7ca-9l7n8" Jul 29 21:48:33.868021 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:33.867926 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6abb428f-5ee7-4186-b81f-d5f0dc03bcb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ceb7ca-9l7n8\" (UID: \"6abb428f-5ee7-4186-b81f-d5f0dc03bcb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ceb7ca-9l7n8" Jul 29 21:48:33.868816 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:33.868790 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6abb428f-5ee7-4186-b81f-d5f0dc03bcb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ceb7ca-9l7n8\" (UID: \"6abb428f-5ee7-4186-b81f-d5f0dc03bcb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ceb7ca-9l7n8" Jul 29 21:48:33.929538 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:33.929491 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ceb7ca-9l7n8" Jul 29 21:48:34.073485 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:34.073447 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ceb7ca-9l7n8"] Jul 29 21:48:34.075158 ip-10-0-137-216 kubenswrapper[2562]: W0729 21:48:34.075132 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abb428f_5ee7_4186_b81f_d5f0dc03bcb8.slice/crio-4ccd2b5085eee62e17fdb81360ee90fe9de4c07b808f53c8ef0bfefa21fb1939 WatchSource:0}: Error finding container 4ccd2b5085eee62e17fdb81360ee90fe9de4c07b808f53c8ef0bfefa21fb1939: Status 404 returned error can't find the container with id 4ccd2b5085eee62e17fdb81360ee90fe9de4c07b808f53c8ef0bfefa21fb1939 Jul 29 21:48:34.946609 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:34.946575 2562 generic.go:356] "Generic (PLEG): container finished" podID="6abb428f-5ee7-4186-b81f-d5f0dc03bcb8" containerID="97c70c9393e2cc4624449750d9008a4a75a669d694c7c8ff947aaaf87c716dff" exitCode=0 Jul 29 21:48:34.947021 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:34.946659 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ceb7ca-9l7n8" event={"ID":"6abb428f-5ee7-4186-b81f-d5f0dc03bcb8","Type":"ContainerDied","Data":"97c70c9393e2cc4624449750d9008a4a75a669d694c7c8ff947aaaf87c716dff"} Jul 29 21:48:34.947021 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:34.946714 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ceb7ca-9l7n8" event={"ID":"6abb428f-5ee7-4186-b81f-d5f0dc03bcb8","Type":"ContainerStarted","Data":"4ccd2b5085eee62e17fdb81360ee90fe9de4c07b808f53c8ef0bfefa21fb1939"} Jul 29 21:48:36.100389 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:36.100350 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ceb7ca-9l7n8" Jul 29 21:48:36.192033 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:36.191990 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6abb428f-5ee7-4186-b81f-d5f0dc03bcb8-maas-api-service-ca\") pod \"6abb428f-5ee7-4186-b81f-d5f0dc03bcb8\" (UID: \"6abb428f-5ee7-4186-b81f-d5f0dc03bcb8\") " Jul 29 21:48:36.192379 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:36.192355 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6abb428f-5ee7-4186-b81f-d5f0dc03bcb8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6abb428f-5ee7-4186-b81f-d5f0dc03bcb8" (UID: "6abb428f-5ee7-4186-b81f-d5f0dc03bcb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:48:36.293366 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:36.293275 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6abb428f-5ee7-4186-b81f-d5f0dc03bcb8-maas-api-service-ca\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:48:36.962035 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:36.961999 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ceb7ca-9l7n8" Jul 29 21:48:36.962222 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:36.962053 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ceb7ca-9l7n8" event={"ID":"6abb428f-5ee7-4186-b81f-d5f0dc03bcb8","Type":"ContainerDied","Data":"4ccd2b5085eee62e17fdb81360ee90fe9de4c07b808f53c8ef0bfefa21fb1939"} Jul 29 21:48:36.962222 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:36.962104 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ccd2b5085eee62e17fdb81360ee90fe9de4c07b808f53c8ef0bfefa21fb1939" Jul 29 21:48:44.170076 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:44.170035 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld"] Jul 29 21:48:44.170628 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:44.170297 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld" podUID="b35c2e71-0617-4eb3-bc90-3c1423edcdcb" containerName="payload-processing" containerID="cri-o://c50376ed92779193456c62b83bb3125b7da812964d9403840bd371e82ea737af" gracePeriod=30 Jul 29 21:48:44.311956 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:44.311898 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ceb7ca-9l7n8"] Jul 29 21:48:44.314562 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:44.314534 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ceb7ca-9l7n8"] Jul 29 21:48:44.436764 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:44.436740 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld" Jul 29 21:48:44.574895 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:44.574856 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b35c2e71-0617-4eb3-bc90-3c1423edcdcb-plugins-config-volume\") pod \"b35c2e71-0617-4eb3-bc90-3c1423edcdcb\" (UID: \"b35c2e71-0617-4eb3-bc90-3c1423edcdcb\") " Jul 29 21:48:44.575142 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:44.574965 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fwmv\" (UniqueName: \"kubernetes.io/projected/b35c2e71-0617-4eb3-bc90-3c1423edcdcb-kube-api-access-2fwmv\") pod \"b35c2e71-0617-4eb3-bc90-3c1423edcdcb\" (UID: \"b35c2e71-0617-4eb3-bc90-3c1423edcdcb\") " Jul 29 21:48:44.575396 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:44.575360 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b35c2e71-0617-4eb3-bc90-3c1423edcdcb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b35c2e71-0617-4eb3-bc90-3c1423edcdcb" (UID: "b35c2e71-0617-4eb3-bc90-3c1423edcdcb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:48:44.577458 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:44.577433 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35c2e71-0617-4eb3-bc90-3c1423edcdcb-kube-api-access-2fwmv" (OuterVolumeSpecName: "kube-api-access-2fwmv") pod "b35c2e71-0617-4eb3-bc90-3c1423edcdcb" (UID: "b35c2e71-0617-4eb3-bc90-3c1423edcdcb"). InnerVolumeSpecName "kube-api-access-2fwmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:48:44.676054 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:44.675962 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fwmv\" (UniqueName: \"kubernetes.io/projected/b35c2e71-0617-4eb3-bc90-3c1423edcdcb-kube-api-access-2fwmv\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:48:44.676054 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:44.675991 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b35c2e71-0617-4eb3-bc90-3c1423edcdcb-plugins-config-volume\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:48:45.012472 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:45.012371 2562 generic.go:356] "Generic (PLEG): container finished" podID="b35c2e71-0617-4eb3-bc90-3c1423edcdcb" containerID="c50376ed92779193456c62b83bb3125b7da812964d9403840bd371e82ea737af" exitCode=0 Jul 29 21:48:45.012472 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:45.012427 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld" Jul 29 21:48:45.012683 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:45.012469 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld" event={"ID":"b35c2e71-0617-4eb3-bc90-3c1423edcdcb","Type":"ContainerDied","Data":"c50376ed92779193456c62b83bb3125b7da812964d9403840bd371e82ea737af"} Jul 29 21:48:45.012683 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:45.012536 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld" event={"ID":"b35c2e71-0617-4eb3-bc90-3c1423edcdcb","Type":"ContainerDied","Data":"8bdcc56f14da94b71e1ebaa5001be93cbc467f644e6926fb7d260a51c51b2a63"} Jul 29 21:48:45.012683 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:45.012572 2562 scope.go:117] "RemoveContainer" containerID="c50376ed92779193456c62b83bb3125b7da812964d9403840bd371e82ea737af" Jul 29 21:48:45.023071 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:45.023052 2562 scope.go:117] "RemoveContainer" containerID="c50376ed92779193456c62b83bb3125b7da812964d9403840bd371e82ea737af" Jul 29 21:48:45.023330 ip-10-0-137-216 kubenswrapper[2562]: E0729 21:48:45.023312 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c50376ed92779193456c62b83bb3125b7da812964d9403840bd371e82ea737af\": container with ID starting with c50376ed92779193456c62b83bb3125b7da812964d9403840bd371e82ea737af not found: ID does not exist" containerID="c50376ed92779193456c62b83bb3125b7da812964d9403840bd371e82ea737af" Jul 29 21:48:45.023371 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:45.023338 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c50376ed92779193456c62b83bb3125b7da812964d9403840bd371e82ea737af"} err="failed to get container status \"c50376ed92779193456c62b83bb3125b7da812964d9403840bd371e82ea737af\": rpc error: code = NotFound desc = could not find container \"c50376ed92779193456c62b83bb3125b7da812964d9403840bd371e82ea737af\": container with ID starting with c50376ed92779193456c62b83bb3125b7da812964d9403840bd371e82ea737af not found: ID does not exist" Jul 29 21:48:45.034805 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:45.034780 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld"] Jul 29 21:48:45.038478 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:45.038455 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ceb7ca-585589f8cd-vbbld"] Jul 29 21:48:45.557890 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:45.557853 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6abb428f-5ee7-4186-b81f-d5f0dc03bcb8" path="/var/lib/kubelet/pods/6abb428f-5ee7-4186-b81f-d5f0dc03bcb8/volumes" Jul 29 21:48:45.558322 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:48:45.558274 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b35c2e71-0617-4eb3-bc90-3c1423edcdcb" path="/var/lib/kubelet/pods/b35c2e71-0617-4eb3-bc90-3c1423edcdcb/volumes" Jul 29 21:50:14.543141 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:14.543108 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25008f-fn9wx"] Jul 29 21:50:14.543571 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:14.543541 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6abb428f-5ee7-4186-b81f-d5f0dc03bcb8" containerName="revoke-keys" Jul 29 21:50:14.543571 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:14.543553 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6abb428f-5ee7-4186-b81f-d5f0dc03bcb8" containerName="revoke-keys" Jul 29 21:50:14.543571 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:14.543565 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b35c2e71-0617-4eb3-bc90-3c1423edcdcb" containerName="payload-processing" Jul 29 21:50:14.543571 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:14.543571 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b35c2e71-0617-4eb3-bc90-3c1423edcdcb" containerName="payload-processing" Jul 29 21:50:14.543711 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:14.543635 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b35c2e71-0617-4eb3-bc90-3c1423edcdcb" containerName="payload-processing" Jul 29 21:50:14.543711 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:14.543649 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6abb428f-5ee7-4186-b81f-d5f0dc03bcb8" containerName="revoke-keys" Jul 29 21:50:14.546658 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:14.546640 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25008f-fn9wx" Jul 29 21:50:14.548673 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:14.548648 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\"" Jul 29 21:50:14.554150 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:14.554122 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25008f-fn9wx"] Jul 29 21:50:14.661333 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:14.661297 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/257111e8-194b-46c6-b267-3e160b3db9d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-25008f-fn9wx\" (UID: \"257111e8-194b-46c6-b267-3e160b3db9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25008f-fn9wx" Jul 29 21:50:14.762409 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:14.762365 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/257111e8-194b-46c6-b267-3e160b3db9d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-25008f-fn9wx\" (UID: \"257111e8-194b-46c6-b267-3e160b3db9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25008f-fn9wx" Jul 29 21:50:14.763043 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:14.763023 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/257111e8-194b-46c6-b267-3e160b3db9d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-25008f-fn9wx\" (UID: \"257111e8-194b-46c6-b267-3e160b3db9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25008f-fn9wx" Jul 29 21:50:14.867237 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:14.867193 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25008f-fn9wx" Jul 29 21:50:15.022195 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:15.022164 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25008f-fn9wx"] Jul 29 21:50:15.023110 ip-10-0-137-216 kubenswrapper[2562]: W0729 21:50:15.023086 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257111e8_194b_46c6_b267_3e160b3db9d2.slice/crio-f24bae27d8a0cf08d5e7d5157126c6bc1668e158ca13270774a9a606e38a412c WatchSource:0}: Error finding container f24bae27d8a0cf08d5e7d5157126c6bc1668e158ca13270774a9a606e38a412c: Status 404 returned error can't find the container with id f24bae27d8a0cf08d5e7d5157126c6bc1668e158ca13270774a9a606e38a412c Jul 29 21:50:15.578781 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:15.578743 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25008f-fn9wx" event={"ID":"257111e8-194b-46c6-b267-3e160b3db9d2","Type":"ContainerStarted","Data":"44692225e530a9afb21afcaa4c581add17c36b602f99906aaf1d06c53ae8a70e"} Jul 29 21:50:15.579411 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:15.578806 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25008f-fn9wx" event={"ID":"257111e8-194b-46c6-b267-3e160b3db9d2","Type":"ContainerStarted","Data":"f24bae27d8a0cf08d5e7d5157126c6bc1668e158ca13270774a9a606e38a412c"} Jul 29 21:50:15.595990 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:15.595905 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25008f-fn9wx" podStartSLOduration=1.59588168 podStartE2EDuration="1.59588168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:50:15.593459266 +0000 UTC m=+3832.464562735" watchObservedRunningTime="2026-07-29 21:50:15.59588168 +0000 UTC m=+3832.466985148" Jul 29 21:50:16.585957 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:16.585902 2562 generic.go:356] "Generic (PLEG): container finished" podID="257111e8-194b-46c6-b267-3e160b3db9d2" containerID="44692225e530a9afb21afcaa4c581add17c36b602f99906aaf1d06c53ae8a70e" exitCode=0 Jul 29 21:50:16.586430 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:16.585954 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25008f-fn9wx" event={"ID":"257111e8-194b-46c6-b267-3e160b3db9d2","Type":"ContainerDied","Data":"44692225e530a9afb21afcaa4c581add17c36b602f99906aaf1d06c53ae8a70e"} Jul 29 21:50:17.728531 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:17.728507 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25008f-fn9wx" Jul 29 21:50:17.792064 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:17.792031 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/257111e8-194b-46c6-b267-3e160b3db9d2-maas-api-service-ca\") pod \"257111e8-194b-46c6-b267-3e160b3db9d2\" (UID: \"257111e8-194b-46c6-b267-3e160b3db9d2\") " Jul 29 21:50:17.792396 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:17.792374 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257111e8-194b-46c6-b267-3e160b3db9d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "257111e8-194b-46c6-b267-3e160b3db9d2" (UID: "257111e8-194b-46c6-b267-3e160b3db9d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:50:17.893010 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:17.892892 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/257111e8-194b-46c6-b267-3e160b3db9d2-maas-api-service-ca\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 21:50:18.599156 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:18.599114 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25008f-fn9wx" Jul 29 21:50:18.599156 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:18.599149 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25008f-fn9wx" event={"ID":"257111e8-194b-46c6-b267-3e160b3db9d2","Type":"ContainerDied","Data":"f24bae27d8a0cf08d5e7d5157126c6bc1668e158ca13270774a9a606e38a412c"} Jul 29 21:50:18.599379 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:18.599199 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f24bae27d8a0cf08d5e7d5157126c6bc1668e158ca13270774a9a606e38a412c" Jul 29 21:50:30.287202 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:30.287150 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25008f-fn9wx"] Jul 29 21:50:30.290371 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:30.290343 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25008f-fn9wx"] Jul 29 21:50:31.557627 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:50:31.557595 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="257111e8-194b-46c6-b267-3e160b3db9d2" path="/var/lib/kubelet/pods/257111e8-194b-46c6-b267-3e160b3db9d2/volumes" Jul 29 21:51:10.711848 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:10.711815 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2"] Jul 29 21:51:10.712458 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:10.712425 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="257111e8-194b-46c6-b267-3e160b3db9d2" containerName="revoke-keys" Jul 29 21:51:10.712458 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:10.712458 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="257111e8-194b-46c6-b267-3e160b3db9d2" containerName="revoke-keys" Jul 29 21:51:10.712597 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:10.712562 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="257111e8-194b-46c6-b267-3e160b3db9d2" containerName="revoke-keys" Jul 29 21:51:10.715972 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:10.715951 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2" Jul 29 21:51:10.718055 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:10.718028 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-ac47f7\"" Jul 29 21:51:10.718166 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:10.718136 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-ac47f7-dockercfg-bpbx9\"" Jul 29 21:51:10.726302 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:10.726277 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2"] Jul 29 21:51:10.806646 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:10.806598 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c0482e-ba54-442e-bdef-34ba173ddcae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2\" (UID: \"22c0482e-ba54-442e-bdef-34ba173ddcae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2" Jul 29 21:51:10.806833 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:10.806657 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg28x\" (UniqueName: \"kubernetes.io/projected/22c0482e-ba54-442e-bdef-34ba173ddcae-kube-api-access-wg28x\") pod \"payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2\" (UID: \"22c0482e-ba54-442e-bdef-34ba173ddcae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2" Jul 29 21:51:10.908068 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:10.908023 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg28x\" (UniqueName: \"kubernetes.io/projected/22c0482e-ba54-442e-bdef-34ba173ddcae-kube-api-access-wg28x\") pod \"payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2\" (UID: \"22c0482e-ba54-442e-bdef-34ba173ddcae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2" Jul 29 21:51:10.908287 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:10.908183 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c0482e-ba54-442e-bdef-34ba173ddcae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2\" (UID: \"22c0482e-ba54-442e-bdef-34ba173ddcae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2" Jul 29 21:51:10.908931 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:10.908891 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c0482e-ba54-442e-bdef-34ba173ddcae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2\" (UID: \"22c0482e-ba54-442e-bdef-34ba173ddcae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2" Jul 29 21:51:10.915656 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:10.915632 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg28x\" (UniqueName: \"kubernetes.io/projected/22c0482e-ba54-442e-bdef-34ba173ddcae-kube-api-access-wg28x\") pod \"payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2\" (UID: \"22c0482e-ba54-442e-bdef-34ba173ddcae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2" Jul 29 21:51:11.028810 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:11.028719 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2" Jul 29 21:51:11.169126 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:11.169094 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2"] Jul 29 21:51:11.170359 ip-10-0-137-216 kubenswrapper[2562]: W0729 21:51:11.170332 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c0482e_ba54_442e_bdef_34ba173ddcae.slice/crio-44e5e09406f61d1e83b24a409793f1ae826a5697616a4e6b70015e1ee04962b9 WatchSource:0}: Error finding container 44e5e09406f61d1e83b24a409793f1ae826a5697616a4e6b70015e1ee04962b9: Status 404 returned error can't find the container with id 44e5e09406f61d1e83b24a409793f1ae826a5697616a4e6b70015e1ee04962b9 Jul 29 21:51:11.949770 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:11.949736 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2" event={"ID":"22c0482e-ba54-442e-bdef-34ba173ddcae","Type":"ContainerStarted","Data":"9af73fcd31c802d01a4fa9f580e70c5d0e29cfe162bdd6bb96ce37b2f9cce41b"} Jul 29 21:51:11.950170 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:11.949797 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2" event={"ID":"22c0482e-ba54-442e-bdef-34ba173ddcae","Type":"ContainerStarted","Data":"44e5e09406f61d1e83b24a409793f1ae826a5697616a4e6b70015e1ee04962b9"} Jul 29 21:51:11.950170 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:11.949856 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2" Jul 29 21:51:11.965975 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:11.965912 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2" podStartSLOduration=1.965895319 podStartE2EDuration="1.965895319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:51:11.96350754 +0000 UTC m=+3888.834611008" watchObservedRunningTime="2026-07-29 21:51:11.965895319 +0000 UTC m=+3888.836998786" Jul 29 21:51:22.959108 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:22.959074 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2" Jul 29 21:51:23.982290 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:23.982168 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:51:24.011449 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:24.011421 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:51:24.230727 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:24.230693 2562 scope.go:117] "RemoveContainer" containerID="ee580e5ebcaccde240520a19d82f90c8a13808373c41eb50ee3a3d8fe5c2e57a" Jul 29 21:51:24.239810 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:51:24.239787 2562 scope.go:117] "RemoveContainer" containerID="aa48d0e0dd3f10e03d4daa29eccc760416d00c6228a4854ccbdb9bc3d732c6b6" Jul 29 21:54:24.288062 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:54:24.287979 2562 scope.go:117] "RemoveContainer" containerID="e97d1b6314e3b0c93789bdf9c9c74e48f7757a2d34968e1711451ea635bc9eaa" Jul 29 21:55:24.311654 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:55:24.311572 2562 scope.go:117] "RemoveContainer" containerID="97c70c9393e2cc4624449750d9008a4a75a669d694c7c8ff947aaaf87c716dff" Jul 29 21:56:24.024478 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:56:24.024376 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:56:24.053599 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:56:24.053576 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 21:56:24.335842 ip-10-0-137-216 kubenswrapper[2562]: I0729 21:56:24.335805 2562 scope.go:117] "RemoveContainer" containerID="44692225e530a9afb21afcaa4c581add17c36b602f99906aaf1d06c53ae8a70e" Jul 29 22:00:00.140542 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:00.140483 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756040-vbrpn"] Jul 29 22:00:00.144988 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:00.144964 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b69e52-29756040-gz8p2"] Jul 29 22:00:00.145099 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:00.145057 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756040-vbrpn" Jul 29 22:00:00.147614 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:00.147586 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\"" Jul 29 22:00:00.149138 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:00.149102 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ac47f7-29756040-fhvpd"] Jul 29 22:00:00.149494 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:00.149470 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b69e52-29756040-gz8p2" Jul 29 22:00:00.153044 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:00.153019 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ac47f7-29756040-fhvpd"] Jul 29 22:00:00.153163 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:00.153114 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ac47f7-29756040-fhvpd" Jul 29 22:00:00.155632 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:00.155368 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756040-vbrpn"] Jul 29 22:00:00.159073 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:00.159040 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b69e52-29756040-gz8p2"] Jul 29 22:00:00.163201 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:00.163181 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756040-vbrpn" Jul 29 22:00:00.171149 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:00.171127 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b69e52-29756040-gz8p2" Jul 29 22:00:00.210207 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:00.210162 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ac47f7-29756040-fhvpd" Jul 29 22:00:00.400562 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:00.400524 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ac47f7-29756040-fhvpd"] Jul 29 22:00:00.401388 ip-10-0-137-216 kubenswrapper[2562]: W0729 22:00:00.401359 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafcabbc_9b48_42ef_badf_3fd27690347d.slice/crio-4347217ff92c723222b0b486c0dd251b2edee424fd14b7c95b3b8bc93510ebcb WatchSource:0}: Error finding container 4347217ff92c723222b0b486c0dd251b2edee424fd14b7c95b3b8bc93510ebcb: Status 404 returned error can't find the container with id 4347217ff92c723222b0b486c0dd251b2edee424fd14b7c95b3b8bc93510ebcb Jul 29 22:00:00.550182 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:00.550153 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b69e52-29756040-gz8p2"] Jul 29 22:00:00.551425 ip-10-0-137-216 kubenswrapper[2562]: W0729 22:00:00.551401 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee40a4b_2d7b_4099_999c_998e820f8045.slice/crio-8b592e9c50b1deafdd1fdc2db2f7a0035140da6612dba3c5f68c1d3918779257 WatchSource:0}: Error finding container 8b592e9c50b1deafdd1fdc2db2f7a0035140da6612dba3c5f68c1d3918779257: Status 404 returned error can't find the container with id 8b592e9c50b1deafdd1fdc2db2f7a0035140da6612dba3c5f68c1d3918779257 Jul 29 22:00:00.575162 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:00.575133 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756040-vbrpn"] Jul 29 22:00:00.591266 ip-10-0-137-216 kubenswrapper[2562]: W0729 22:00:00.591234 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d9e460_bbca_4900_95a0_4bfb08b2186c.slice/crio-aa6eefe392d026ab4f368b3de52a3ab7931a329738f07cf07fadcbf779351a6f WatchSource:0}: Error finding container aa6eefe392d026ab4f368b3de52a3ab7931a329738f07cf07fadcbf779351a6f: Status 404 returned error can't find the container with id aa6eefe392d026ab4f368b3de52a3ab7931a329738f07cf07fadcbf779351a6f Jul 29 22:00:01.114753 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:01.114712 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b69e52-29756040-gz8p2" event={"ID":"0ee40a4b-2d7b-4099-999c-998e820f8045","Type":"ContainerStarted","Data":"6f0842409bca52840dc096ac8f737a170d9a9fa069493743affcd7b4ea85e538"} Jul 29 22:00:01.114982 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:01.114774 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b69e52-29756040-gz8p2" event={"ID":"0ee40a4b-2d7b-4099-999c-998e820f8045","Type":"ContainerStarted","Data":"8b592e9c50b1deafdd1fdc2db2f7a0035140da6612dba3c5f68c1d3918779257"} Jul 29 22:00:01.116315 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:01.116289 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ac47f7-29756040-fhvpd" event={"ID":"cafcabbc-9b48-42ef-badf-3fd27690347d","Type":"ContainerStarted","Data":"4d1233461ebf1b660024da23fdfabbe1ead1427f3bd58a93ad4c444377cae8e0"} Jul 29 22:00:01.116429 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:01.116337 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ac47f7-29756040-fhvpd" event={"ID":"cafcabbc-9b48-42ef-badf-3fd27690347d","Type":"ContainerStarted","Data":"4347217ff92c723222b0b486c0dd251b2edee424fd14b7c95b3b8bc93510ebcb"} Jul 29 22:00:01.117787 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:01.117766 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756040-vbrpn" event={"ID":"b4d9e460-bbca-4900-95a0-4bfb08b2186c","Type":"ContainerStarted","Data":"cc13bb0c1314ecc8e54f0d5397140184ece9a18ae4297671636e392fe9b94ac3"} Jul 29 22:00:01.117907 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:01.117809 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756040-vbrpn" event={"ID":"b4d9e460-bbca-4900-95a0-4bfb08b2186c","Type":"ContainerStarted","Data":"aa6eefe392d026ab4f368b3de52a3ab7931a329738f07cf07fadcbf779351a6f"} Jul 29 22:00:01.131452 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:01.131403 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b69e52-29756040-gz8p2" podStartSLOduration=1.131388276 podStartE2EDuration="1.131388276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:00:01.128318973 +0000 UTC m=+4417.999422440" watchObservedRunningTime="2026-07-29 22:00:01.131388276 +0000 UTC m=+4418.002491786" Jul 29 22:00:01.142775 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:01.142712 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756040-vbrpn" podStartSLOduration=1.142696902 podStartE2EDuration="1.142696902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:00:01.140199358 +0000 UTC m=+4418.011302819" watchObservedRunningTime="2026-07-29 22:00:01.142696902 +0000 UTC m=+4418.013800428" Jul 29 22:00:01.156085 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:01.156029 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ac47f7-29756040-fhvpd" podStartSLOduration=1.156015187 podStartE2EDuration="1.156015187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:00:01.153596367 +0000 UTC m=+4418.024699838" watchObservedRunningTime="2026-07-29 22:00:01.156015187 +0000 UTC m=+4418.027118653" Jul 29 22:00:02.125686 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:02.125652 2562 generic.go:356] "Generic (PLEG): container finished" podID="cafcabbc-9b48-42ef-badf-3fd27690347d" containerID="4d1233461ebf1b660024da23fdfabbe1ead1427f3bd58a93ad4c444377cae8e0" exitCode=0 Jul 29 22:00:02.125871 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:02.125750 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ac47f7-29756040-fhvpd" event={"ID":"cafcabbc-9b48-42ef-badf-3fd27690347d","Type":"ContainerDied","Data":"4d1233461ebf1b660024da23fdfabbe1ead1427f3bd58a93ad4c444377cae8e0"} Jul 29 22:00:02.127276 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:02.127254 2562 generic.go:356] "Generic (PLEG): container finished" podID="b4d9e460-bbca-4900-95a0-4bfb08b2186c" containerID="cc13bb0c1314ecc8e54f0d5397140184ece9a18ae4297671636e392fe9b94ac3" exitCode=0 Jul 29 22:00:02.127413 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:02.127330 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756040-vbrpn" event={"ID":"b4d9e460-bbca-4900-95a0-4bfb08b2186c","Type":"ContainerDied","Data":"cc13bb0c1314ecc8e54f0d5397140184ece9a18ae4297671636e392fe9b94ac3"} Jul 29 22:00:02.128840 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:02.128821 2562 generic.go:356] "Generic (PLEG): container finished" podID="0ee40a4b-2d7b-4099-999c-998e820f8045" containerID="6f0842409bca52840dc096ac8f737a170d9a9fa069493743affcd7b4ea85e538" exitCode=0 Jul 29 22:00:02.129030 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:02.128844 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b69e52-29756040-gz8p2" event={"ID":"0ee40a4b-2d7b-4099-999c-998e820f8045","Type":"ContainerDied","Data":"6f0842409bca52840dc096ac8f737a170d9a9fa069493743affcd7b4ea85e538"} Jul 29 22:00:03.273561 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:03.273519 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b69e52-29756040-gz8p2" Jul 29 22:00:03.318222 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:03.318194 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756040-vbrpn" Jul 29 22:00:03.360393 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:03.360365 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ac47f7-29756040-fhvpd" Jul 29 22:00:04.142747 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:04.142716 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ac47f7-29756040-fhvpd" Jul 29 22:00:04.142978 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:04.142757 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ac47f7-29756040-fhvpd" event={"ID":"cafcabbc-9b48-42ef-badf-3fd27690347d","Type":"ContainerDied","Data":"4347217ff92c723222b0b486c0dd251b2edee424fd14b7c95b3b8bc93510ebcb"} Jul 29 22:00:04.142978 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:04.142838 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4347217ff92c723222b0b486c0dd251b2edee424fd14b7c95b3b8bc93510ebcb" Jul 29 22:00:04.144335 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:04.144315 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756040-vbrpn" Jul 29 22:00:04.144442 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:04.144342 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756040-vbrpn" event={"ID":"b4d9e460-bbca-4900-95a0-4bfb08b2186c","Type":"ContainerDied","Data":"aa6eefe392d026ab4f368b3de52a3ab7931a329738f07cf07fadcbf779351a6f"} Jul 29 22:00:04.144442 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:04.144382 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa6eefe392d026ab4f368b3de52a3ab7931a329738f07cf07fadcbf779351a6f" Jul 29 22:00:04.145806 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:04.145784 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b69e52-29756040-gz8p2" event={"ID":"0ee40a4b-2d7b-4099-999c-998e820f8045","Type":"ContainerDied","Data":"8b592e9c50b1deafdd1fdc2db2f7a0035140da6612dba3c5f68c1d3918779257"} Jul 29 22:00:04.145928 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:04.145805 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b69e52-29756040-gz8p2" Jul 29 22:00:04.145928 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:00:04.145844 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b592e9c50b1deafdd1fdc2db2f7a0035140da6612dba3c5f68c1d3918779257" Jul 29 22:01:24.064348 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:01:24.064321 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 22:01:24.095041 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:01:24.095015 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 22:02:47.966462 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:47.966374 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr"] Jul 29 22:02:47.966971 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:47.966806 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ee40a4b-2d7b-4099-999c-998e820f8045" containerName="cleanup" Jul 29 22:02:47.966971 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:47.966816 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ee40a4b-2d7b-4099-999c-998e820f8045" containerName="cleanup" Jul 29 22:02:47.966971 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:47.966826 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4d9e460-bbca-4900-95a0-4bfb08b2186c" containerName="cleanup" Jul 29 22:02:47.966971 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:47.966831 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4d9e460-bbca-4900-95a0-4bfb08b2186c" containerName="cleanup" Jul 29 22:02:47.966971 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:47.966842 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cafcabbc-9b48-42ef-badf-3fd27690347d" containerName="cleanup" Jul 29 22:02:47.966971 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:47.966848 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cafcabbc-9b48-42ef-badf-3fd27690347d" containerName="cleanup" Jul 29 22:02:47.966971 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:47.966936 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ee40a4b-2d7b-4099-999c-998e820f8045" containerName="cleanup" Jul 29 22:02:47.966971 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:47.966947 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cafcabbc-9b48-42ef-badf-3fd27690347d" containerName="cleanup" Jul 29 22:02:47.966971 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:47.966953 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4d9e460-bbca-4900-95a0-4bfb08b2186c" containerName="cleanup" Jul 29 22:02:47.970166 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:47.970149 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr" Jul 29 22:02:47.972132 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:47.972111 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\"" Jul 29 22:02:47.976367 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:47.976348 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr"] Jul 29 22:02:48.055006 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:48.054961 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24de8a7a-5d17-45e9-acde-fc9dbc99eda4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr\" (UID: \"24de8a7a-5d17-45e9-acde-fc9dbc99eda4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr" Jul 29 22:02:48.156561 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:48.156526 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24de8a7a-5d17-45e9-acde-fc9dbc99eda4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr\" (UID: \"24de8a7a-5d17-45e9-acde-fc9dbc99eda4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr" Jul 29 22:02:48.157265 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:48.157237 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24de8a7a-5d17-45e9-acde-fc9dbc99eda4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr\" (UID: \"24de8a7a-5d17-45e9-acde-fc9dbc99eda4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr" Jul 29 22:02:48.283570 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:48.283483 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr" Jul 29 22:02:48.630390 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:48.630356 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr"] Jul 29 22:02:48.631277 ip-10-0-137-216 kubenswrapper[2562]: W0729 22:02:48.631250 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24de8a7a_5d17_45e9_acde_fc9dbc99eda4.slice/crio-85a35621a4e26cec9f01c29dbb4321ef34decf1a8752e68667a17580ee75db6d WatchSource:0}: Error finding container 85a35621a4e26cec9f01c29dbb4321ef34decf1a8752e68667a17580ee75db6d: Status 404 returned error can't find the container with id 85a35621a4e26cec9f01c29dbb4321ef34decf1a8752e68667a17580ee75db6d Jul 29 22:02:49.170252 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:49.170147 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr" event={"ID":"24de8a7a-5d17-45e9-acde-fc9dbc99eda4","Type":"ContainerStarted","Data":"4d5cb69ccaac4f13cb55e2ca8dc0a5636bfce7b3f4db8cc85b0bfabde1f97c35"} Jul 29 22:02:49.170252 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:49.170214 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr" event={"ID":"24de8a7a-5d17-45e9-acde-fc9dbc99eda4","Type":"ContainerStarted","Data":"85a35621a4e26cec9f01c29dbb4321ef34decf1a8752e68667a17580ee75db6d"} Jul 29 22:02:49.187256 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:49.187203 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr" podStartSLOduration=2.187187603 podStartE2EDuration="2.187187603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:02:49.184569866 +0000 UTC m=+4586.055673332" watchObservedRunningTime="2026-07-29 22:02:49.187187603 +0000 UTC m=+4586.058291069" Jul 29 22:02:50.178532 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:50.178498 2562 generic.go:356] "Generic (PLEG): container finished" podID="24de8a7a-5d17-45e9-acde-fc9dbc99eda4" containerID="4d5cb69ccaac4f13cb55e2ca8dc0a5636bfce7b3f4db8cc85b0bfabde1f97c35" exitCode=0 Jul 29 22:02:50.178911 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:50.178561 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr" event={"ID":"24de8a7a-5d17-45e9-acde-fc9dbc99eda4","Type":"ContainerDied","Data":"4d5cb69ccaac4f13cb55e2ca8dc0a5636bfce7b3f4db8cc85b0bfabde1f97c35"} Jul 29 22:02:51.324047 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:51.324021 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr" Jul 29 22:02:51.389788 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:51.389751 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24de8a7a-5d17-45e9-acde-fc9dbc99eda4-maas-api-service-ca\") pod \"24de8a7a-5d17-45e9-acde-fc9dbc99eda4\" (UID: \"24de8a7a-5d17-45e9-acde-fc9dbc99eda4\") " Jul 29 22:02:51.390142 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:51.390117 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24de8a7a-5d17-45e9-acde-fc9dbc99eda4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "24de8a7a-5d17-45e9-acde-fc9dbc99eda4" (UID: "24de8a7a-5d17-45e9-acde-fc9dbc99eda4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 22:02:51.491149 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:51.491064 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24de8a7a-5d17-45e9-acde-fc9dbc99eda4-maas-api-service-ca\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 22:02:52.192263 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:52.192206 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr" event={"ID":"24de8a7a-5d17-45e9-acde-fc9dbc99eda4","Type":"ContainerDied","Data":"85a35621a4e26cec9f01c29dbb4321ef34decf1a8752e68667a17580ee75db6d"} Jul 29 22:02:52.192454 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:52.192287 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85a35621a4e26cec9f01c29dbb4321ef34decf1a8752e68667a17580ee75db6d" Jul 29 22:02:52.192454 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:52.192220 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr" Jul 29 22:02:58.409101 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:58.409059 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b69e52-29756040-gz8p2"] Jul 29 22:02:58.412659 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:58.412633 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b69e52-29756040-gz8p2"] Jul 29 22:02:58.593098 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:58.593061 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr"] Jul 29 22:02:58.596929 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:58.596875 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b69e52-4bhnr"] Jul 29 22:02:59.562934 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:59.562889 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ee40a4b-2d7b-4099-999c-998e820f8045" path="/var/lib/kubelet/pods/0ee40a4b-2d7b-4099-999c-998e820f8045/volumes" Jul 29 22:02:59.563437 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:02:59.563383 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24de8a7a-5d17-45e9-acde-fc9dbc99eda4" path="/var/lib/kubelet/pods/24de8a7a-5d17-45e9-acde-fc9dbc99eda4/volumes" Jul 29 22:03:05.465041 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:05.464999 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd"] Jul 29 22:03:05.465449 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:05.465436 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24de8a7a-5d17-45e9-acde-fc9dbc99eda4" containerName="revoke-keys" Jul 29 22:03:05.465492 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:05.465450 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="24de8a7a-5d17-45e9-acde-fc9dbc99eda4" containerName="revoke-keys" Jul 29 22:03:05.465566 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:05.465557 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="24de8a7a-5d17-45e9-acde-fc9dbc99eda4" containerName="revoke-keys" Jul 29 22:03:05.469763 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:05.469744 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd" Jul 29 22:03:05.471804 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:05.471774 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-frnc6\"" Jul 29 22:03:05.482311 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:05.482272 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd"] Jul 29 22:03:05.527074 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:05.527029 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e1f7d8e-e5ca-4135-936b-097a787f9189-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd\" (UID: \"8e1f7d8e-e5ca-4135-936b-097a787f9189\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd" Jul 29 22:03:05.628258 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:05.628223 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e1f7d8e-e5ca-4135-936b-097a787f9189-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd\" (UID: \"8e1f7d8e-e5ca-4135-936b-097a787f9189\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd" Jul 29 22:03:05.628994 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:05.628974 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e1f7d8e-e5ca-4135-936b-097a787f9189-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd\" (UID: \"8e1f7d8e-e5ca-4135-936b-097a787f9189\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd" Jul 29 22:03:05.786943 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:05.786833 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd" Jul 29 22:03:05.934471 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:05.932150 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd"] Jul 29 22:03:06.280646 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:06.280607 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd" event={"ID":"8e1f7d8e-e5ca-4135-936b-097a787f9189","Type":"ContainerStarted","Data":"9d36e2653b9336cb4406eb396124a21da961442baa60333829f6bb830d758f60"} Jul 29 22:03:06.280818 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:06.280668 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd" event={"ID":"8e1f7d8e-e5ca-4135-936b-097a787f9189","Type":"ContainerStarted","Data":"a4688ee36685d1913502ec6c4144596afe754a15e857c0888e9c80f881eb88a6"} Jul 29 22:03:06.297419 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:06.297369 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd" podStartSLOduration=1.297354819 podStartE2EDuration="1.297354819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:03:06.293873848 +0000 UTC m=+4603.164977327" watchObservedRunningTime="2026-07-29 22:03:06.297354819 +0000 UTC m=+4603.168458644" Jul 29 22:03:07.288241 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:07.288203 2562 generic.go:356] "Generic (PLEG): container finished" podID="8e1f7d8e-e5ca-4135-936b-097a787f9189" containerID="9d36e2653b9336cb4406eb396124a21da961442baa60333829f6bb830d758f60" exitCode=0 Jul 29 22:03:07.288619 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:07.288288 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd" event={"ID":"8e1f7d8e-e5ca-4135-936b-097a787f9189","Type":"ContainerDied","Data":"9d36e2653b9336cb4406eb396124a21da961442baa60333829f6bb830d758f60"} Jul 29 22:03:08.439160 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:08.439131 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd" Jul 29 22:03:08.557219 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:08.557192 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e1f7d8e-e5ca-4135-936b-097a787f9189-maas-api-service-ca\") pod \"8e1f7d8e-e5ca-4135-936b-097a787f9189\" (UID: \"8e1f7d8e-e5ca-4135-936b-097a787f9189\") " Jul 29 22:03:08.557545 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:08.557520 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1f7d8e-e5ca-4135-936b-097a787f9189-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8e1f7d8e-e5ca-4135-936b-097a787f9189" (UID: "8e1f7d8e-e5ca-4135-936b-097a787f9189"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 22:03:08.658177 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:08.658132 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e1f7d8e-e5ca-4135-936b-097a787f9189-maas-api-service-ca\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 22:03:09.303169 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:09.303133 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd" event={"ID":"8e1f7d8e-e5ca-4135-936b-097a787f9189","Type":"ContainerDied","Data":"a4688ee36685d1913502ec6c4144596afe754a15e857c0888e9c80f881eb88a6"} Jul 29 22:03:09.303169 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:09.303161 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd" Jul 29 22:03:09.303389 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:09.303183 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4688ee36685d1913502ec6c4144596afe754a15e857c0888e9c80f881eb88a6" Jul 29 22:03:15.887295 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:15.887259 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ac47f7-29756040-fhvpd"] Jul 29 22:03:15.891900 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:15.891866 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ac47f7-29756040-fhvpd"] Jul 29 22:03:15.926640 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:15.926603 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2"] Jul 29 22:03:15.926892 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:15.926863 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2" podUID="22c0482e-ba54-442e-bdef-34ba173ddcae" containerName="payload-pre-processing" containerID="cri-o://9af73fcd31c802d01a4fa9f580e70c5d0e29cfe162bdd6bb96ce37b2f9cce41b" gracePeriod=30 Jul 29 22:03:16.061402 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:16.061366 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd"] Jul 29 22:03:16.066515 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:16.066483 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ac47f7-ln4rd"] Jul 29 22:03:16.205804 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:16.205766 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2" Jul 29 22:03:16.339266 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:16.339233 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wg28x\" (UniqueName: \"kubernetes.io/projected/22c0482e-ba54-442e-bdef-34ba173ddcae-kube-api-access-wg28x\") pod \"22c0482e-ba54-442e-bdef-34ba173ddcae\" (UID: \"22c0482e-ba54-442e-bdef-34ba173ddcae\") " Jul 29 22:03:16.339460 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:16.339322 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c0482e-ba54-442e-bdef-34ba173ddcae-plugins-config-volume\") pod \"22c0482e-ba54-442e-bdef-34ba173ddcae\" (UID: \"22c0482e-ba54-442e-bdef-34ba173ddcae\") " Jul 29 22:03:16.340343 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:16.340290 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c0482e-ba54-442e-bdef-34ba173ddcae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22c0482e-ba54-442e-bdef-34ba173ddcae" (UID: "22c0482e-ba54-442e-bdef-34ba173ddcae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 22:03:16.342062 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:16.340619 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c0482e-ba54-442e-bdef-34ba173ddcae-plugins-config-volume\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 22:03:16.343381 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:16.342206 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c0482e-ba54-442e-bdef-34ba173ddcae-kube-api-access-wg28x" (OuterVolumeSpecName: "kube-api-access-wg28x") pod "22c0482e-ba54-442e-bdef-34ba173ddcae" (UID: "22c0482e-ba54-442e-bdef-34ba173ddcae"). InnerVolumeSpecName "kube-api-access-wg28x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 22:03:16.353738 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:16.353712 2562 generic.go:356] "Generic (PLEG): container finished" podID="22c0482e-ba54-442e-bdef-34ba173ddcae" containerID="9af73fcd31c802d01a4fa9f580e70c5d0e29cfe162bdd6bb96ce37b2f9cce41b" exitCode=0 Jul 29 22:03:16.353842 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:16.353761 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2" Jul 29 22:03:16.353842 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:16.353799 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2" event={"ID":"22c0482e-ba54-442e-bdef-34ba173ddcae","Type":"ContainerDied","Data":"9af73fcd31c802d01a4fa9f580e70c5d0e29cfe162bdd6bb96ce37b2f9cce41b"} Jul 29 22:03:16.353951 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:16.353874 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2" event={"ID":"22c0482e-ba54-442e-bdef-34ba173ddcae","Type":"ContainerDied","Data":"44e5e09406f61d1e83b24a409793f1ae826a5697616a4e6b70015e1ee04962b9"} Jul 29 22:03:16.353951 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:16.353900 2562 scope.go:117] "RemoveContainer" containerID="9af73fcd31c802d01a4fa9f580e70c5d0e29cfe162bdd6bb96ce37b2f9cce41b" Jul 29 22:03:16.363605 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:16.363590 2562 scope.go:117] "RemoveContainer" containerID="9af73fcd31c802d01a4fa9f580e70c5d0e29cfe162bdd6bb96ce37b2f9cce41b" Jul 29 22:03:16.363829 ip-10-0-137-216 kubenswrapper[2562]: E0729 22:03:16.363810 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af73fcd31c802d01a4fa9f580e70c5d0e29cfe162bdd6bb96ce37b2f9cce41b\": container with ID starting with 9af73fcd31c802d01a4fa9f580e70c5d0e29cfe162bdd6bb96ce37b2f9cce41b not found: ID does not exist" containerID="9af73fcd31c802d01a4fa9f580e70c5d0e29cfe162bdd6bb96ce37b2f9cce41b" Jul 29 22:03:16.363883 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:16.363836 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af73fcd31c802d01a4fa9f580e70c5d0e29cfe162bdd6bb96ce37b2f9cce41b"} err="failed to get container status \"9af73fcd31c802d01a4fa9f580e70c5d0e29cfe162bdd6bb96ce37b2f9cce41b\": rpc error: code = NotFound desc = could not find container \"9af73fcd31c802d01a4fa9f580e70c5d0e29cfe162bdd6bb96ce37b2f9cce41b\": container with ID starting with 9af73fcd31c802d01a4fa9f580e70c5d0e29cfe162bdd6bb96ce37b2f9cce41b not found: ID does not exist" Jul 29 22:03:16.375372 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:16.375348 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2"] Jul 29 22:03:16.381980 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:16.381953 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-ac47f7-5b466858d-bhsc2"] Jul 29 22:03:16.441846 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:16.441761 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg28x\" (UniqueName: \"kubernetes.io/projected/22c0482e-ba54-442e-bdef-34ba173ddcae-kube-api-access-wg28x\") on node \"ip-10-0-137-216.ec2.internal\" DevicePath \"\"" Jul 29 22:03:17.558261 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:17.558223 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22c0482e-ba54-442e-bdef-34ba173ddcae" path="/var/lib/kubelet/pods/22c0482e-ba54-442e-bdef-34ba173ddcae/volumes" Jul 29 22:03:17.558637 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:17.558610 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e1f7d8e-e5ca-4135-936b-097a787f9189" path="/var/lib/kubelet/pods/8e1f7d8e-e5ca-4135-936b-097a787f9189/volumes" Jul 29 22:03:17.558867 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:17.558854 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cafcabbc-9b48-42ef-badf-3fd27690347d" path="/var/lib/kubelet/pods/cafcabbc-9b48-42ef-badf-3fd27690347d/volumes" Jul 29 22:03:28.897215 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:28.897183 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bjtgx_5a870fec-1f88-41c2-8bf6-e239cbac331c/server/0.log" Jul 29 22:03:29.018676 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:29.018647 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-6s7ss_1eeae0db-d5e9-4447-a622-8415e6b733a3/manager/0.log" Jul 29 22:03:29.128900 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:29.128871 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dbc94cc7-dxxkm_85f40ff7-a510-4ece-bec3-665fac570daa/copy-manifests/0.log" Jul 29 22:03:29.140634 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:29.140607 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dbc94cc7-dxxkm_85f40ff7-a510-4ece-bec3-665fac570daa/manager/0.log" Jul 29 22:03:29.735034 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:29.735005 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-848d99f5fb-d85qg_34716149-c1ff-4198-b465-149d7c0e8d14/maas-api/0.log" Jul 29 22:03:30.079578 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:30.079554 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756040-vbrpn_b4d9e460-bbca-4900-95a0-4bfb08b2186c/cleanup/0.log" Jul 29 22:03:30.190331 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:30.190305 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-db5sk_c833efa9-365d-4d5b-b4c1-2a114f077fb2/postgres/0.log" Jul 29 22:03:30.970342 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:30.970315 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9_9a5db510-214d-43a6-bfdd-a6d3ad45ca01/util/0.log" Jul 29 22:03:30.975679 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:30.975655 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9_9a5db510-214d-43a6-bfdd-a6d3ad45ca01/pull/0.log" Jul 29 22:03:30.981341 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:30.981322 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9_9a5db510-214d-43a6-bfdd-a6d3ad45ca01/extract/0.log" Jul 29 22:03:31.099162 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:31.099130 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp_2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c/pull/0.log" Jul 29 22:03:31.104518 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:31.104477 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp_2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c/extract/0.log" Jul 29 22:03:31.109184 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:31.109164 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp_2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c/util/0.log" Jul 29 22:03:31.222975 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:31.222878 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4_c9eb5e4c-6906-441e-bfdf-a1f42e282365/util/0.log" Jul 29 22:03:31.228771 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:31.228748 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4_c9eb5e4c-6906-441e-bfdf-a1f42e282365/pull/0.log" Jul 29 22:03:31.234577 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:31.234559 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4_c9eb5e4c-6906-441e-bfdf-a1f42e282365/extract/0.log" Jul 29 22:03:31.351409 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:31.351380 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt_84af0a1a-df6a-405a-9b11-0fcc1f090256/util/0.log" Jul 29 22:03:31.356500 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:31.356480 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt_84af0a1a-df6a-405a-9b11-0fcc1f090256/pull/0.log" Jul 29 22:03:31.361612 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:31.361594 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt_84af0a1a-df6a-405a-9b11-0fcc1f090256/extract/0.log" Jul 29 22:03:31.722001 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:31.721974 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-fzznx_38178c0f-2d1b-435b-8c09-944bc9f8152d/manager/0.log" Jul 29 22:03:31.958576 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:31.958544 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-rg4zg_df3a395b-ae6b-4641-93f0-8dd49fff2a81/registry-server/0.log" Jul 29 22:03:32.083509 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:32.083473 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-6c2rk_ee4c41be-8d9e-4366-90dd-8d972beeeb21/manager/0.log" Jul 29 22:03:32.779735 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:32.779703 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bqn6f_6bffcc3c-447e-46fa-bdce-d5c5af921ead/discovery/0.log" Jul 29 22:03:32.998827 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:32.998796 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b585ddb49-xvld7_e97f8c98-53dd-4308-b78b-35672457368b/kube-auth-proxy/0.log" Jul 29 22:03:33.114800 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:33.114761 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2229g_fa5cd0c8-a7b7-45e0-9015-de4ad64b8219/istio-proxy/1.log" Jul 29 22:03:33.227014 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:33.226984 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-cdblw_372a884b-b391-42b9-9299-cb2173cbabe1/payload-pre-processing/0.log" Jul 29 22:03:33.337410 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:33.337367 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-mmtk9_24fdea1b-1074-47cf-9b52-d39f228fa43b/payload-processing/0.log" Jul 29 22:03:33.442526 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:33.442439 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f46dc85cf-9562c_d56bff06-5480-4bcc-bf5b-dbfd2372f9cd/router/0.log" Jul 29 22:03:33.793032 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:33.792949 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-9wl7p_5cd41d02-77ed-4d6b-9cb1-d6a5e5fda162/main/0.log" Jul 29 22:03:34.148898 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:34.148871 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm9s4t_862f377a-5b72-466b-8185-0cda71bbda3e/main/0.log" Jul 29 22:03:34.262617 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:34.262568 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-jdkhw_3f9f2a65-b3f3-4b62-a674-0d305110b511/main/0.log" Jul 29 22:03:43.023714 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:43.023676 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fhgt5_ba4aa585-620a-4c88-8c46-09b343186c70/global-pull-secret-syncer/0.log" Jul 29 22:03:43.251282 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:43.251253 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mc9jj_d172a83f-d86e-4d6b-85ae-792d777676c5/konnectivity-agent/0.log" Jul 29 22:03:43.298205 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:43.298126 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-216.ec2.internal_4119ef05f02339ab6df3a952dfb37eab/haproxy/0.log" Jul 29 22:03:47.594299 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:47.594267 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9_9a5db510-214d-43a6-bfdd-a6d3ad45ca01/extract/0.log" Jul 29 22:03:47.619566 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:47.619537 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9_9a5db510-214d-43a6-bfdd-a6d3ad45ca01/util/0.log" Jul 29 22:03:47.640288 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:47.640258 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759svzd9_9a5db510-214d-43a6-bfdd-a6d3ad45ca01/pull/0.log" Jul 29 22:03:47.672552 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:47.672523 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp_2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c/extract/0.log" Jul 29 22:03:47.694946 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:47.694906 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp_2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c/util/0.log" Jul 29 22:03:47.718859 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:47.718831 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf2vp_2bd388a8-9f43-42a2-b1c1-0c45a0e77f9c/pull/0.log" Jul 29 22:03:47.751174 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:47.751147 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4_c9eb5e4c-6906-441e-bfdf-a1f42e282365/extract/0.log" Jul 29 22:03:47.773702 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:47.773674 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4_c9eb5e4c-6906-441e-bfdf-a1f42e282365/util/0.log" Jul 29 22:03:47.796355 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:47.796328 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6gn4_c9eb5e4c-6906-441e-bfdf-a1f42e282365/pull/0.log" Jul 29 22:03:47.827977 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:47.827909 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt_84af0a1a-df6a-405a-9b11-0fcc1f090256/extract/0.log" Jul 29 22:03:47.847874 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:47.847811 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt_84af0a1a-df6a-405a-9b11-0fcc1f090256/util/0.log" Jul 29 22:03:47.868878 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:47.868857 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef197xxt_84af0a1a-df6a-405a-9b11-0fcc1f090256/pull/0.log" Jul 29 22:03:48.568212 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:48.568186 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-fzznx_38178c0f-2d1b-435b-8c09-944bc9f8152d/manager/0.log" Jul 29 22:03:48.646072 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:48.646039 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-rg4zg_df3a395b-ae6b-4641-93f0-8dd49fff2a81/registry-server/0.log" Jul 29 22:03:48.716434 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:48.716401 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-6c2rk_ee4c41be-8d9e-4366-90dd-8d972beeeb21/manager/0.log" Jul 29 22:03:50.561074 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:50.561050 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f05ba6f3-359e-4f72-82da-edae6eb4df21/alertmanager/0.log" Jul 29 22:03:50.587886 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:50.587857 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f05ba6f3-359e-4f72-82da-edae6eb4df21/config-reloader/0.log" Jul 29 22:03:50.612516 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:50.612479 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f05ba6f3-359e-4f72-82da-edae6eb4df21/kube-rbac-proxy-web/0.log" Jul 29 22:03:50.647956 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:50.647913 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f05ba6f3-359e-4f72-82da-edae6eb4df21/kube-rbac-proxy/0.log" Jul 29 22:03:50.669425 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:50.669400 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f05ba6f3-359e-4f72-82da-edae6eb4df21/kube-rbac-proxy-metric/0.log" Jul 29 22:03:50.694700 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:50.694648 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f05ba6f3-359e-4f72-82da-edae6eb4df21/prom-label-proxy/0.log" Jul 29 22:03:50.721122 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:50.721095 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f05ba6f3-359e-4f72-82da-edae6eb4df21/init-config-reloader/0.log" Jul 29 22:03:50.766504 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:50.766419 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-brnnx_1f181cb2-5f41-425b-97d5-6057df483f86/cluster-monitoring-operator/0.log" Jul 29 22:03:50.790423 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:50.790397 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-rqpwg_2ffa2c61-2436-4157-8edc-a5f360c28d8b/kube-state-metrics/0.log" Jul 29 22:03:50.811142 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:50.811114 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-rqpwg_2ffa2c61-2436-4157-8edc-a5f360c28d8b/kube-rbac-proxy-main/0.log" Jul 29 22:03:50.831015 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:50.830994 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-rqpwg_2ffa2c61-2436-4157-8edc-a5f360c28d8b/kube-rbac-proxy-self/0.log" Jul 29 22:03:50.858563 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:50.858538 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-577c88c6f7-qb7xj_14d57622-e026-4e5e-8608-16450670ad62/metrics-server/0.log" Jul 29 22:03:50.935429 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:50.935399 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7l48v_26cbdcfd-b96e-405f-912c-db5b721a7f91/node-exporter/0.log" Jul 29 22:03:50.957405 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:50.957380 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7l48v_26cbdcfd-b96e-405f-912c-db5b721a7f91/kube-rbac-proxy/0.log" Jul 29 22:03:50.977931 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:50.977906 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7l48v_26cbdcfd-b96e-405f-912c-db5b721a7f91/init-textfile/0.log" Jul 29 22:03:52.966179 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:52.966147 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-q7z24_30987044-0cc1-4e94-91d9-4712292608be/networking-console-plugin/0.log" Jul 29 22:03:53.221421 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.221337 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw"] Jul 29 22:03:53.221786 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.221766 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c0482e-ba54-442e-bdef-34ba173ddcae" containerName="payload-pre-processing" Jul 29 22:03:53.221786 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.221780 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c0482e-ba54-442e-bdef-34ba173ddcae" containerName="payload-pre-processing" Jul 29 22:03:53.221994 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.221798 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e1f7d8e-e5ca-4135-936b-097a787f9189" containerName="revoke-keys" Jul 29 22:03:53.221994 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.221807 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e1f7d8e-e5ca-4135-936b-097a787f9189" containerName="revoke-keys" Jul 29 22:03:53.221994 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.221912 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e1f7d8e-e5ca-4135-936b-097a787f9189" containerName="revoke-keys" Jul 29 22:03:53.221994 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.221950 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c0482e-ba54-442e-bdef-34ba173ddcae" containerName="payload-pre-processing" Jul 29 22:03:53.225527 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.225512 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" Jul 29 22:03:53.227570 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.227550 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-crrnx\"/\"kube-root-ca.crt\"" Jul 29 22:03:53.228007 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.227991 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-crrnx\"/\"default-dockercfg-rf2bz\"" Jul 29 22:03:53.228066 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.228013 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-crrnx\"/\"openshift-service-ca.crt\"" Jul 29 22:03:53.238439 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.238417 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw"] Jul 29 22:03:53.290235 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.290202 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c81638d9-a59b-44f6-92d0-fc2644e1e3c6-sys\") pod \"perf-node-gather-daemonset-bwfbw\" (UID: \"c81638d9-a59b-44f6-92d0-fc2644e1e3c6\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" Jul 29 22:03:53.290235 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.290236 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c81638d9-a59b-44f6-92d0-fc2644e1e3c6-proc\") pod \"perf-node-gather-daemonset-bwfbw\" (UID: \"c81638d9-a59b-44f6-92d0-fc2644e1e3c6\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" Jul 29 22:03:53.290485 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.290297 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c81638d9-a59b-44f6-92d0-fc2644e1e3c6-podres\") pod \"perf-node-gather-daemonset-bwfbw\" (UID: \"c81638d9-a59b-44f6-92d0-fc2644e1e3c6\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" Jul 29 22:03:53.290485 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.290317 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r6kw\" (UniqueName: \"kubernetes.io/projected/c81638d9-a59b-44f6-92d0-fc2644e1e3c6-kube-api-access-9r6kw\") pod \"perf-node-gather-daemonset-bwfbw\" (UID: \"c81638d9-a59b-44f6-92d0-fc2644e1e3c6\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" Jul 29 22:03:53.290485 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.290394 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c81638d9-a59b-44f6-92d0-fc2644e1e3c6-lib-modules\") pod \"perf-node-gather-daemonset-bwfbw\" (UID: \"c81638d9-a59b-44f6-92d0-fc2644e1e3c6\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" Jul 29 22:03:53.391744 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.391708 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c81638d9-a59b-44f6-92d0-fc2644e1e3c6-sys\") pod \"perf-node-gather-daemonset-bwfbw\" (UID: \"c81638d9-a59b-44f6-92d0-fc2644e1e3c6\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" Jul 29 22:03:53.391744 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.391751 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c81638d9-a59b-44f6-92d0-fc2644e1e3c6-proc\") pod \"perf-node-gather-daemonset-bwfbw\" (UID: \"c81638d9-a59b-44f6-92d0-fc2644e1e3c6\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" Jul 29 22:03:53.392059 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.391800 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c81638d9-a59b-44f6-92d0-fc2644e1e3c6-podres\") pod \"perf-node-gather-daemonset-bwfbw\" (UID: \"c81638d9-a59b-44f6-92d0-fc2644e1e3c6\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" Jul 29 22:03:53.392059 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.391826 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r6kw\" (UniqueName: \"kubernetes.io/projected/c81638d9-a59b-44f6-92d0-fc2644e1e3c6-kube-api-access-9r6kw\") pod \"perf-node-gather-daemonset-bwfbw\" (UID: \"c81638d9-a59b-44f6-92d0-fc2644e1e3c6\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" Jul 29 22:03:53.392059 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.391839 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c81638d9-a59b-44f6-92d0-fc2644e1e3c6-sys\") pod \"perf-node-gather-daemonset-bwfbw\" (UID: \"c81638d9-a59b-44f6-92d0-fc2644e1e3c6\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" Jul 29 22:03:53.392059 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.391915 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c81638d9-a59b-44f6-92d0-fc2644e1e3c6-proc\") pod \"perf-node-gather-daemonset-bwfbw\" (UID: \"c81638d9-a59b-44f6-92d0-fc2644e1e3c6\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" Jul 29 22:03:53.392059 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.391911 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c81638d9-a59b-44f6-92d0-fc2644e1e3c6-lib-modules\") pod \"perf-node-gather-daemonset-bwfbw\" (UID: \"c81638d9-a59b-44f6-92d0-fc2644e1e3c6\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" Jul 29 22:03:53.392059 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.392004 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c81638d9-a59b-44f6-92d0-fc2644e1e3c6-podres\") pod \"perf-node-gather-daemonset-bwfbw\" (UID: \"c81638d9-a59b-44f6-92d0-fc2644e1e3c6\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" Jul 29 22:03:53.392059 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.392027 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c81638d9-a59b-44f6-92d0-fc2644e1e3c6-lib-modules\") pod \"perf-node-gather-daemonset-bwfbw\" (UID: \"c81638d9-a59b-44f6-92d0-fc2644e1e3c6\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" Jul 29 22:03:53.399362 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.399333 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r6kw\" (UniqueName: \"kubernetes.io/projected/c81638d9-a59b-44f6-92d0-fc2644e1e3c6-kube-api-access-9r6kw\") pod \"perf-node-gather-daemonset-bwfbw\" (UID: \"c81638d9-a59b-44f6-92d0-fc2644e1e3c6\") " pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" Jul 29 22:03:53.535841 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.535747 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" Jul 29 22:03:53.637358 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.637331 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/1.log" Jul 29 22:03:53.642813 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.642784 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8v2vf_bf9e0545-985e-47df-af42-4453c1e24937/console-operator/2.log" Jul 29 22:03:53.670952 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:53.670895 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw"] Jul 29 22:03:53.672441 ip-10-0-137-216 kubenswrapper[2562]: W0729 22:03:53.672413 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc81638d9_a59b_44f6_92d0_fc2644e1e3c6.slice/crio-3efc5ac94fc258f8344e89ae4c7cd83b8ad871b858e5a0171a5684cbe48737d4 WatchSource:0}: Error finding container 3efc5ac94fc258f8344e89ae4c7cd83b8ad871b858e5a0171a5684cbe48737d4: Status 404 returned error can't find the container with id 3efc5ac94fc258f8344e89ae4c7cd83b8ad871b858e5a0171a5684cbe48737d4 Jul 29 22:03:54.151737 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:54.151703 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6f6fc64598-vdwfp_73b01475-72a3-4c4e-b91a-20aa5c6cc175/console/0.log" Jul 29 22:03:54.591100 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:54.591059 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" event={"ID":"c81638d9-a59b-44f6-92d0-fc2644e1e3c6","Type":"ContainerStarted","Data":"8a864962ce003f6cac119e3ba86be2deaaf606334846f0e7fd853f9fb331d26d"} Jul 29 22:03:54.591365 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:54.591122 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" event={"ID":"c81638d9-a59b-44f6-92d0-fc2644e1e3c6","Type":"ContainerStarted","Data":"3efc5ac94fc258f8344e89ae4c7cd83b8ad871b858e5a0171a5684cbe48737d4"} Jul 29 22:03:54.591365 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:54.591150 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" Jul 29 22:03:54.606792 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:54.606740 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" podStartSLOduration=1.606724646 podStartE2EDuration="1.606724646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:03:54.605234953 +0000 UTC m=+4651.476338431" watchObservedRunningTime="2026-07-29 22:03:54.606724646 +0000 UTC m=+4651.477828126" Jul 29 22:03:54.752911 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:54.752878 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-q9s7m_c4622d90-8ccb-4ee8-80a8-6a0a0d4e552d/volume-data-source-validator/0.log" Jul 29 22:03:55.868715 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:55.868682 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t89sj_9749b441-c685-4fa6-8044-791cae15b712/dns/0.log" Jul 29 22:03:55.889035 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:55.888988 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t89sj_9749b441-c685-4fa6-8044-791cae15b712/kube-rbac-proxy/0.log" Jul 29 22:03:55.935298 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:55.935274 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l4txq_43dd2f3d-02e2-47ba-b20a-4365c2c6f45b/dns-node-resolver/0.log" Jul 29 22:03:56.567158 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:56.567116 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p87w8_e6d19b51-bb38-465c-ae75-cfa9d195bb72/node-ca/0.log" Jul 29 22:03:57.664061 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:57.664022 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bqn6f_6bffcc3c-447e-46fa-bdce-d5c5af921ead/discovery/0.log" Jul 29 22:03:57.710802 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:57.710770 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b585ddb49-xvld7_e97f8c98-53dd-4308-b78b-35672457368b/kube-auth-proxy/0.log" Jul 29 22:03:57.748071 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:57.748023 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2229g_fa5cd0c8-a7b7-45e0-9015-de4ad64b8219/istio-proxy/0.log" Jul 29 22:03:57.772807 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:57.772771 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2229g_fa5cd0c8-a7b7-45e0-9015-de4ad64b8219/istio-proxy/1.log" Jul 29 22:03:57.804046 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:57.804005 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-cdblw_372a884b-b391-42b9-9299-cb2173cbabe1/payload-pre-processing/0.log" Jul 29 22:03:57.833096 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:57.833059 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-mmtk9_24fdea1b-1074-47cf-9b52-d39f228fa43b/payload-processing/0.log" Jul 29 22:03:57.857266 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:57.857231 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f46dc85cf-9562c_d56bff06-5480-4bcc-bf5b-dbfd2372f9cd/router/0.log" Jul 29 22:03:58.711193 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:58.711162 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zwgsv_43c0ba7b-1895-4952-b45b-87f2983002d1/serve-healthcheck-canary/0.log" Jul 29 22:03:59.197978 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:59.197933 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-9hbmn_a08729b6-2792-466e-b7c4-fb0ae44087e6/insights-operator/0.log" Jul 29 22:03:59.199236 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:59.199210 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-9hbmn_a08729b6-2792-466e-b7c4-fb0ae44087e6/insights-operator/1.log" Jul 29 22:03:59.362232 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:59.362201 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vgs2s_053c21f8-14cd-4244-a80d-9713ee979ca3/kube-rbac-proxy/0.log" Jul 29 22:03:59.382163 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:59.382135 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vgs2s_053c21f8-14cd-4244-a80d-9713ee979ca3/exporter/0.log" Jul 29 22:03:59.403362 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:03:59.403331 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vgs2s_053c21f8-14cd-4244-a80d-9713ee979ca3/extractor/0.log" Jul 29 22:04:00.607843 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:00.607807 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-crrnx/perf-node-gather-daemonset-bwfbw" Jul 29 22:04:02.212573 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:02.212544 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bjtgx_5a870fec-1f88-41c2-8bf6-e239cbac331c/server/0.log" Jul 29 22:04:02.292716 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:02.292683 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-6s7ss_1eeae0db-d5e9-4447-a622-8415e6b733a3/manager/0.log" Jul 29 22:04:02.391883 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:02.391853 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dbc94cc7-dxxkm_85f40ff7-a510-4ece-bec3-665fac570daa/manager/0.log" Jul 29 22:04:02.413206 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:02.413159 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dbc94cc7-dxxkm_85f40ff7-a510-4ece-bec3-665fac570daa/copy-manifests/0.log" Jul 29 22:04:03.922884 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:03.922855 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-8hmzs_b786c126-dc63-4b7c-8dd3-7ecefc1c7f33/openshift-lws-operator/0.log" Jul 29 22:04:04.096315 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:04.096282 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bqn6f_6bffcc3c-447e-46fa-bdce-d5c5af921ead/discovery/0.log" Jul 29 22:04:04.122443 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:04.122399 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b585ddb49-xvld7_e97f8c98-53dd-4308-b78b-35672457368b/kube-auth-proxy/0.log" Jul 29 22:04:04.142999 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:04.142964 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2229g_fa5cd0c8-a7b7-45e0-9015-de4ad64b8219/istio-proxy/0.log" Jul 29 22:04:04.171347 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:04.171314 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2229g_fa5cd0c8-a7b7-45e0-9015-de4ad64b8219/istio-proxy/1.log" Jul 29 22:04:04.198335 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:04.198301 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-cdblw_372a884b-b391-42b9-9299-cb2173cbabe1/payload-pre-processing/0.log" Jul 29 22:04:04.217325 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:04.217294 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-mmtk9_24fdea1b-1074-47cf-9b52-d39f228fa43b/payload-processing/0.log" Jul 29 22:04:04.234955 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:04.234911 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f46dc85cf-9562c_d56bff06-5480-4bcc-bf5b-dbfd2372f9cd/router/0.log" Jul 29 22:04:05.739330 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:05.739246 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bqn6f_6bffcc3c-447e-46fa-bdce-d5c5af921ead/discovery/0.log" Jul 29 22:04:05.770532 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:05.770487 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b585ddb49-xvld7_e97f8c98-53dd-4308-b78b-35672457368b/kube-auth-proxy/0.log" Jul 29 22:04:05.797881 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:05.797847 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2229g_fa5cd0c8-a7b7-45e0-9015-de4ad64b8219/istio-proxy/0.log" Jul 29 22:04:05.826121 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:05.826093 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2229g_fa5cd0c8-a7b7-45e0-9015-de4ad64b8219/istio-proxy/1.log" Jul 29 22:04:05.846083 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:05.846053 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-cdblw_372a884b-b391-42b9-9299-cb2173cbabe1/payload-pre-processing/0.log" Jul 29 22:04:05.868061 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:05.868033 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-mmtk9_24fdea1b-1074-47cf-9b52-d39f228fa43b/payload-processing/0.log" Jul 29 22:04:05.881244 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:05.881224 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f46dc85cf-9562c_d56bff06-5480-4bcc-bf5b-dbfd2372f9cd/router/0.log" Jul 29 22:04:09.223489 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:09.223454 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7_4250508f-31ea-4575-b84e-50a57d1aa159/kube-storage-version-migrator-operator/1.log" Jul 29 22:04:09.224446 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:09.224423 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-7k9v7_4250508f-31ea-4575-b84e-50a57d1aa159/kube-storage-version-migrator-operator/0.log" Jul 29 22:04:10.648049 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:10.648020 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9nwx_a11939fc-63a4-4f61-9de5-5d9bef849715/kube-multus-additional-cni-plugins/0.log" Jul 29 22:04:10.668849 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:10.668824 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9nwx_a11939fc-63a4-4f61-9de5-5d9bef849715/egress-router-binary-copy/0.log" Jul 29 22:04:10.688035 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:10.688011 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9nwx_a11939fc-63a4-4f61-9de5-5d9bef849715/cni-plugins/0.log" Jul 29 22:04:10.707679 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:10.707655 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9nwx_a11939fc-63a4-4f61-9de5-5d9bef849715/bond-cni-plugin/0.log" Jul 29 22:04:10.726375 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:10.726346 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9nwx_a11939fc-63a4-4f61-9de5-5d9bef849715/routeoverride-cni/0.log" Jul 29 22:04:10.747031 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:10.747002 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9nwx_a11939fc-63a4-4f61-9de5-5d9bef849715/whereabouts-cni-bincopy/0.log" Jul 29 22:04:10.776123 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:10.776097 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v9nwx_a11939fc-63a4-4f61-9de5-5d9bef849715/whereabouts-cni/0.log" Jul 29 22:04:10.850031 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:10.850003 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-njm9k_b1661ddb-eba2-4408-a48b-f2edcc9b600e/kube-multus/0.log" Jul 29 22:04:10.870623 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:10.870586 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5sgzv_347e06fd-9f35-47dd-9442-ddd628e5bd35/network-metrics-daemon/0.log" Jul 29 22:04:10.888948 ip-10-0-137-216 kubenswrapper[2562]: I0729 22:04:10.888905 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5sgzv_347e06fd-9f35-47dd-9442-ddd628e5bd35/kube-rbac-proxy/0.log"